how to deduplicate huggingface datasets about deduplicate-text-datasets HOT 7 CLOSED

@TristanThrush I don't think this repo requires you to load everything into memory though. I'll try running Teven's PR and report back.

from deduplicate-text-datasets.

@cakiki we might be able to resolve this issue on the text-dedup side: ChenghaoMou/text-dedup#5

from deduplicate-text-datasets.

This repository does require that the dataset itself fit in memory. Closing because I've now merged #6.

from deduplicate-text-datasets.

A while ago someone sent a PR to do this

#6

But I haven't tested it on the latest code -- maybe you could try that and see if it would work for you?

from deduplicate-text-datasets.

I just had the same question as you @StephennFernandes, and I found this repo which can be installed with pip: https://github.com/ChenghaoMou/text-dedup. It looks like text-dedup has a more plug-and-play version of this repo? I followed the documentation here and used it on a huggingface dataset with embedder.embed_bash(my_hf_dataset["text"])

from deduplicate-text-datasets.

@TristanThrush This only works on datasets that fit in memory though, right? I think my_hf_dataset["text"] loads the entire column into a list. I'm also trying to solve this for a larger-than-memory dataset, have you had any luck with this?

from deduplicate-text-datasets.

@cakiki possibly. I have a machine with 100's of GB of memory, so it hasn't been a problem for me. I think that both this repo and the alternative repo require a lot of memory?

from deduplicate-text-datasets.