pandas-linker runs comparison windows over different sortings of a pandas DataFrame and links the rows via assigned UUIDs. This library does not actually do any duplicate detection. Instead it provides a harness to run your own comparison functions on your data.

This library is meant for datasets of a size where comparing every row with every other is undesirable. Instead you can decide on a sorting order of the DataFrame and only compare every row with every other inside a sliding window.

pip install pandas-linker

Let's say you have a DataFrame like this:

and you want to detect similar rows and mark them as such. Here's how to do that:

from pandas_linker import get_linker


def compare_rows(a, b):
    ''' Example function that decides if two rows represent same entity.'''
    return a['name'] in b['name'] or b['name'] in a['name']

# df is a pandas.DataFrame with a unique index

with get_linker(df, field='uid') as linker:

    print('Comparing in 10 row window sorted by name')
    linker(sort_cols=['name'], window_size=10, cmp=compare_rows)

    print('Comparing in 15 row window sorted by country')
    linker(sort_cols=['country'], window_size=15, cmp=compare_rows)

After running the linker the process is complete

Note that both "Mary" rows in the DataFrame have been identified as representing the same entity and were assigned the same UUID.