The shpank/data
folder contains data from geofabrik.
To make tests run smoothly the .shp
files which are >100MB are moved to shpank/data/big
.
The integration tests then skip them, but we can still run them manually.
hyperfine -w3 "./target/release/parse --shp shpank/data/gis_osm_natural_a_free_1.shp --mode bytes" "./target/release/parse
--shp shpank/data/gis_osm_natural_a_free_1.shp --mode file"
The >GB sized files can take up to 10s to parse. If we had a strategy to multi-thread this could likely be a lot faster.
If we find parsing the .shx
files is fast, we could try to allow interpreting the contents
as jobs.
It might be a good strategy:
gis_osm_water_a_free_1.shp
is 1.3 GiB, whilegis_osm_water_a_free_1.shx
is 14.2 MiB
We could then benchmark doing .shx
-> jobs -> N threads (N from CLI) -> parse.
Each job would only need:
- Knowing which byte offset it should start parsing records from
- Knowing how many bytes it should read, which is all the bytes until the next job's starting offset
If we use an ordered rayon flat map sort of approach we might end up in a good place.
Let's add some tracing integration so we can see how long various parts of code takes.
Showing results in borld
won't be fun unless we parse this data, which tells us that a shape is supposed to be a bus stop or a tree or a road, and things like their names.