PBF error: illegal blob size about osmium-tool HOT 5 OPEN

@joto thank you for your answer.

If you want support for that, please contact me by email and I'll send you my consulting rates.

Ok. If we decide that we definitely must send our propriety data to you, I'll hand-over your consulting fee to my manager.

My suspicion is about can_add function in libosmium. Please see here:
https://github.com/osmcode/libosmium/blob/f88048769c13210ca81efca17668dc57ea64c632/include/osmium/io/detail/pbf_output_format.hpp#L362

In this line return size() < max_used_blob_size;, we compare if the current size of the blob is less than the max allowed blob size, and then we blindly add the entity to the blob, without checking if the entity to be added is not too big to exceed the limit.

In other words, we don't check the size() + sizeOfEntityToBeAdded < max_used_blob_size.

Is my understanding correct from the can_add function?

from osmium-tool.

I can not reproduce this. I used the USA file from Geofabrik and did the steps you describe and the resulting file is fine.

I had to repair the config file you provided, it does not work otherwise. Are you sure the osmium extract command even ran? Maybe the broken Delaware.osm.pbf is left over from an earlier attempt?

The other thing: The Delaware.osm.pbf you seem to have is rather large, 650 MB. Mine is only 19 MB.

from osmium-tool.

The input PBF file that I have for USA is indeed enriched with other data sources. That's why it's bigger. Would it be ok for you if I upload a piece of USA that I'm working with somewhere, and send it to you to reproduce the issue? Shall I send the link privately to [email protected] ? Otherwise I'd violate my company's data privacy.

Also what was the repair that you did on the config file?

Are you sure the osmium extract command even ran? Maybe the broken Delaware.osm.pbf is left over from an earlier attempt?

Yes, I'm pretty sure that osmium extract ran, since I see the output of osmium about extract.

from osmium-tool.

Ah, you should have mentioned that you are working with proprietary data. If you want support for that, please contact me by email and I'll send you my consulting rates.

from osmium-tool.

Side note: as workaround, I changed the max_entities_per_block to 1000 (instead of the default 8000), compiled the osmium + libosmium, and now the PBFs that osmium extract of the custom osmium creates are valid (no longer illegal blob size).

Would you agree that this is a signal that the can_add function does not return always the correct value, as I described above?

from osmium-tool.