Ignore list order about dictdiffer HOT 13 OPEN

Actually, it is not convenient to compare two dicts from json. set is not been supported by json, so I need to convert list to set at first before I diff. Maybe it will be nice to have ignore_order, though I could convert list to set.

from dictdiffer.

@inoks can you use set instead of list if you are not interested in the order of items?

from dictdiffer import diff

first = {'a': {1,2}}
second = {'a': {2,1}}

result = diff(first, second)
assert list(result) == []

from dictdiffer.

@jirikuncar Actually, no. My dict came from xmltodict, which parsed pretty large xml documents.
I ended up creating dict sorting function, which sorts all nested dict and lists, before using diff().

from dictdiffer.

@jirikuncar And i also want duplicate items in list to be detected by diff().

from dictdiffer.

@inoks if you have a working solution for your domain specific problem, can you please close the issue?

from dictdiffer.

@jirikuncar I prefer that issue still be there, just in case someone decide to implement that in the future.

from dictdiffer.

To be honest, I don't think it should be part of diffing library. The same way as you diff two files in UNIX.

$ diff first.txt second.txt

The lines are not sorted unless you explicitly pre-sort them.

$ sort first.txt > sorted_first.txt
$ sort second.txt > sorted_second.txt
$ diff soted_first.txt sorted_second.txt

What if somebody wants only certains keys to be sorted, or provide custom sorting method? I'm worried that the API complexity would explode.

from dictdiffer.

I agree with @jirikuncar. A Python list is an ordered collection, and thus order should be taken into consideration when diff'ing two lists. If order doesn't matter then either a set should be used or the lists should be preprocessed (i.e. sorted) prior to being diff'ed. It will make the algorithm overly complex and likely introduced subtle errors to add this feature.

I understand your issue, but perhaps another solution is to simply create another package with your preprocessor function that relies on diff to do the job? IMHO it's best to keep a algorithm strict and not add all sorts of exceptions into it.

from dictdiffer.

I will agree with both of you, but there already tolerance option existed, so it will be nice to have ignore_order as well.

from dictdiffer.

@inoks the tolerance is there because of limitations in operations with floating points numbers. Two "identical" values can be compared to False when they are calculated in different ways. This problem is not possible to solve with pure preprocessing without rounding values.

Having said that I would really like to keep the package minimalist [1] and let developers to create their own pipelines.

[1] http://www.catb.org/esr/writings/taoup/html/ch01s06.html

from dictdiffer.

I humbly disagree with the previous assessments.

I'm working with abritrary sets of data and tens of thousands of dictionary records, which i do not control and currently the diffing is triggering when some lists are in arbitrary order. It's not always practical to manipulate these dictionaries each time before they're checked, i'd rather not touch them as I may introduce variability in my code. It would be nice if the tool would have a built-in feature that could just ignore the order as long as the content is the same for a given field.

Another library DeepDiff has this feature, but it has issues with some fields of data while this tool does a better job at it.

The set() workaround doesn't work when the objects in a list are not hashable, sorted(list) may work better in those cases. (it's likely deepdiff is using this method internally why i'm seeing issues diffing certain types of lists)

Lastly just because something is bad and inconvenient in unix, it doesn't mean it should be inconvenient in this tool. The unix-example is just sorting one list contained in a file, but these dictionaries are complex nested constructs with arbitrary objects.

from dictdiffer.

@unitysipu, do you want to add/implement sorting hooks for different types?

from dictdiffer.

I wish i had the time, but I can see how this is not always a trivial problem.

Looks like DeepDiff has some kind of deephash function just recently introduced to try to solve some of these issues, but It's not (yet) clear how i can apply that. Anycase, I hope my use case gives more insight to why this feature would be useful in some cases.

Currently for this dataset deepdiff will just reports it ignores the unhashable fields but doesn't prevent diffing the rest, I maybe able to live with that for now, but I need to probably return to this problem when some field of true significance starts exhibiting issues.

from dictdiffer.