[osmium-merge] With 2 input files merge does not clean duplicates about osmium-tool HOT 4 CLOSED

Strictly speaking you are using osmium merge outside the spec if you have duplicates in any of the input files. Osmium only promises to remove duplicates if they appear in different files, not if the duplicates are in the same file.

I never thought about the case you have. It is kind of difficult to understand the difference and explain this to users. And it is easy to fix by removing the special case you mentioned. So I'll fix it.

from osmium-tool.

Seems I was overhasty. This is not so easy to fix. You say it doesn't happen in the three-file case, but that isn't so. Maybe it does in your case, but generally that is not true.

Why do you have files with duplicate entries in the first place?

from osmium-tool.

I merge generated files into a single one. It's not data from the OSM site but data in a OSM compliant format.

The files contain adminstrative boundary of different level.

There is no duplicate data in the same file, but the 2 files contains the same nodes and even the same way, like island for exemple.

In those cases, the output contains all the nodes and ways from the 2 inputs file.

(I use osmconvert to be able to grep inside the pbf files)

extract from input file 1 :

grep 'id="22659884631506728' France/pbfFiles/fra.osm                                     
	<way id="22659884631506728" version="1" timestamp="2017-02-08T09:40:38Z" changeset="1" uid="1" user="Toto">

extract fom input file 2 :

grep 'id="22659884631506728' France/pbfFiles/fraf22.osm
	<way id="22659884631506728" version="1" timestamp="2017-02-08T09:41:39Z" changeset="1" uid="1" user="Toto">

extract from output file :

grep 'id="22659884631506728' France/France2.osm
	<way id="22659884631506728" version="1" timestamp="2017-02-08T09:40:38Z" changeset="1" uid="1" user="Toto">
	<way id="22659884631506728" version="1" timestamp="2017-02-08T09:41:39Z" changeset="1" uid="1" user="Toto">

but if I use a second time the fra.osm.pbf file then the output is :

grep 'id="22659884631506728' France/France3.osm         
	<way id="22659884631506728" version="1" timestamp="2017-02-08T09:41:39Z" changeset="1" uid="1" user="Toto">

My thought : as the timestamp is not the same, the set_union use in the 2 files case does not see the 2 ways as identicals. whereas in the 3 files case no set_union is made.

I'm not a cpp dev, but a java one, i might have miss something :)

from osmium-tool.

If you have two different variants of the same object but with the same id and version than all bets are off. That's not correct data and I can't guarantee any outcome. I'll clarify this in the man page.

The reason you are seeing the different behaviour is probably that one algorithm uses == comparison and the other <. But this is not something you can rely upon.

from osmium-tool.