Comments (8)
From @DGuidi on February 28, 2017 9:44
Any way to compress the result? Germany now give me a shapefile of 70GB (!).
My2Cents: shapefile is a binary format with a well defined standard, so size of file is directly related to size of data. maybe you can (I think):
- simplify your geometries
- create your own "ShapefileZipWriter" that directly generates a zipped archive of a shapefile (if possible)
from nettopologysuite.io.shapefile.
Oh... does BigEndianBinaryWriter need the same performance improvement I did in c6d2ccd? I see some calls to that class's naive WriteIntBE method. Maybe the corresponding reader too...
from nettopologysuite.io.shapefile.
If that doesn't help, could you please provide sample code and maybe a sample serialized RouterDb file so we can look at the same thing? I've got an old serialized routerdb file, but it dates back to the times when Itinero was part of OsmSharp, and I'm guessing you've changed stuff since then, and I really don't want to dedicate CPU time to rerun odp assuming it's still as slow as it was back then.
If I can just get that little bit of help, I'd love to spend time agonizing over this one.
from nettopologysuite.io.shapefile.
From @xivk on February 28, 2017 14:59
I'll try and build a sample application and do some profiling too, stay tuned. :-)
from nettopologysuite.io.shapefile.
I've done some exploration in my branch perf-exploration
:
- e2aba74 stops us from allocating extra on every big-endian write
- 578a508 stops us from flushing the output writer after every feature when we don't have to (i.e., when we're not writing a
.shx
file that needs to know) - aa6953d moves a heap allocation out of a loop
- 294ec76 handles writing the
.shp
,.dbf
, and.shx
(if any) files all in a single scan of the input features (also gets rid of this method's version of the heap allocation I moved out of that other loop) - 9e8e1c1 lets us actually skip writing the
.shx
file if we don't want to (lots of stuff in the code seems to think that it might ever be possible for us not to write this). - baeb13f combines what used to be separate
.Min()
and.Max()
loops into one, for both Z and M ordinates. - 40a6b0d gets rid of another place where we would flush the stream before and after writing each feature; this time, we would just do it in order to throw an exception if there's a bug in our own code, which seems unnecessary (maybe we can bring it back with a property on the writer or something).
from nettopologysuite.io.shapefile.
Related to this issue, if we can get rid of the count required parameter in the header we can use IEnumerables as input:
Maybe it's possible to write this count after all features have been enumerated?
I did notice a huge improvement already because we now seem to only enumerating the collection once instead of twice! 👍 💯
from nettopologysuite.io.shapefile.
write this count after all features have been enumerated
A problem with this is that we would have to seek, which means that we'd start only supporting seekable streams (at least for callers that don't have a count
to pass in, though branching like that adds maintenance cost), which may turn out to be really awkward.
from nettopologysuite.io.shapefile.
Yes, I also considered that, but it's a pretty big show stopper for some usecases not to be able 'stream' the data into the writer.
Maybe we should just stop using shapefiles ;-)
Anyway, for now, I count features in another way before writing so I'm not blocked on this or something like that.
from nettopologysuite.io.shapefile.
Related Issues (20)
- System.NotSupportedException : Memory stream is not expandable. HOT 3
- Inconsistency determining and writing Int32 in the DBaseFileHeader HOT 1
- System.OverflowException when reading empty header HOT 1
- Creating a Dbase column with a length larger 254 (1 byte) results in strange behavior HOT 7
- String values are trimmed when read using DbaseFileReader HOT 1
- Null Values in Dbase file HOT 1
- Maybe a little bug about class DbaseFileHeader's encoding ? HOT 11
- Geometry with only Z aren't supported HOT 4
- Cannot read a polygon with a hole from a ShapeFile HOT 3
- Write shape data to MemoryStreams -> MemoryStream not expandable HOT 2
- How to update existing shape file HOT 2
- DbaseFileHeader.DetectEncoding method should be modified HOT 3
- Max "M" value is always set to double.PositiveInfinity HOT 5
- Can't encoding UTF-8 error HOT 2
- GDB missing HOT 8
- Publish release with ExternallyManagedStreamProvider HOT 1
- NullReferenceException in ShapeDataReader HOT 5
- What is the license of NetTopologySuite.IO.ShapeFile? HOT 25
- Discontinued HOT 2
- DbaseFileWriter DBF write Geometry issue with the Geometry type "Polygon" and "Multipolygon" HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nettopologysuite.io.shapefile.