When we have the paper somewhere on arxiv, put a link to it in this repo so people using this can cite the paper

I was looking at this codebase and encountered this bit:
https://github.com/bigscience-workshop/data-preparation/tree/main/sourcing/code_dataset#code-dataset-sourcing

The query to create the dataset can be found in query.sql. After creation the dataset was preprocessed with processing.py. Note that there is a bug in the script that filters only for GPL licenses instead of filtering them out. There are instructions to remove the bug but it is left there for reproducibility.

This leads me to believe that the code here is meant to be used and investigated "as is" and without modification.
Is this repo primarily meant for reproducibility?

If i wanted to improve and extend it for an independent Dataset building project, should I fork it or work from a branch?

When thomas/re_organize branch is merged, we need rename the slurm scripts inside repos/data-preparation/sourcing/cc_pseudo_crawl/seeds_batch_1 by:

download_warc.slurm  -> 01_download_warc.slurm    
download_warc_trial_4.slurm -> 02_download_warc_trial_4.slurm    
download_warc_trial_5.slurm -> 03_download_warc_trial_5.slurm
download_warc_too_big.slurm -> 04_download_warc_too_big.slurm
redownload_warc.slurm -> 05_redownload_warc.slurm
check_errors_in_dataset.slurm -> 06_check_errors_in_dataset.slurm  
preprocess_warc.slurm -> 08_preprocess_warc.slurm  
extract_text_and_html_metadata.slurm -> 09_extract_text_and_html_metadata.slurm  
shard_by_seed_id.slurm -> 10_shard_by_seed_id.slurm
merge_seed_shards.slurm -> 11_merge_seed_shards.slurm
shard_and_compress.slurm -> 12_shard_and_compress.slurm

Then we still need to find which of the following files have been used in the end divide_in_subshards.slurm or divide_in_subshards_1000.slurm (step 7)

EDIT: divide_in_subshards.slurm is the step 7 and divide_in_subshards_1000.slurm is reality done in step 10

I tried to download English part of Roots these days.
According to the paper, there are 484,953,009,124 bytes of English data.
However, after downloading all roots-related datasets on huggingface by filtering, I found there is only about 43.8 GB data.
I wonder how to explain the difference?
Are those huggingface datasets only a subset of Roots?
Are those huggingface datasets processed Roots so that the quantity shrinks from 480 GB to 43.8 GB?

when I run get_list_of_datasets.py ,i get "Number of datasets 0" ,this result means that there is no data in huggingface using this author,how can i get the data. Thanks for your work.

Hi! Kudos to the author for an end-to-end piepline for cleaning and filtering a large corpus. I was working with main_filtering.py and was trying to change the parameter values in parameters_filtering.py, hoping to increase/decrease the no. of documents that were being removed out. But I observe no changes.

1. I have english dataset so I set parameters_filtering_en, and I have experimented with the given values and some modifications in 1/more conditions and cutoffs.
2. I have also tried out parameters_filtering_default where I do observe changes in documents being filtered out. The no. was different from those in parameters_filtering_en.

• The parameters_filtering_default has some error. I modified languages_id.py to account for "defualt" as langauge but used flagged_/stop_words of english language.

3. Within parameters_filtering_default or parameters_filtering_en, when parameter values are changed no changes are observed in no. of documents or documents which are getting removed.

Kindly review the code and let me know the solutions. Also let me know if I'm missing something.

Thank You!

Thanks for your amazing codebase!

I find that the link of Deduplication Report in preprocessing/training/01b_oscar_cleaning_and_filtering/deduplicate/README.md is not accessible.

Could you please update it?

Especially in README files, at the very end, convert every link pointing to another repo (for example data_tooling) to this repo when possible

We extend our gratitude to the authors of this repository! Your documentation and code have greatly benefited the community.

We have used this repo in building the data processing pipeline tool SailCraft.
It consists of four stages: initial data cleaning, near deduplication, exact deduplication, and a second round of data cleaning.

Many thanks for your contribution for open research! And we welcome the developers to try SailCraft.

Thanks for your helpful codebase!

I am a bit confused about stop words filtering.
The release code removes the document, if its stop words ratio below the certain cutoff.

I am wondering which one is more useful in your practice.
Thanks in advance!

Which version of simhash is used in the project, and why is the output of simhash.find_all() method always an empty list?

We need to write a bunch of readmes on how we used each tools.