bigcode-project / bigcode-analysis Goto Github PK

Provide results on large dataset with different near deduplication methods:

1. minhash + lsh
2. simhash
3. any relevant methods

Details to be included:

• tokenization method
• method parameters
• hardware
• memory usage
• time
• duplication results, examples

The link to python-all-license is broken in the data analysis readme.

As we extend deduplication to a wide range of languages, what tokenization method to use will have an impact on the final results.

The current script uses a simple regex and uni-gram to perform minhash calculation. What are the consequences using a different configuration?

The current script building clusters of duplicates, but there are cases it might yield unwanted results:

When doc B is clustered under doc A's name, another doc C can also be clustered into B's name (AB, BC, C!~A), thus when we are deleting non "extreme"s from each cluster, we could end up having both A and B kept in the results.

A better way to delete duplicates is to find community within each connected components. This is used in https://github.com/src-d/gemini.

Loading multi_query_experiments.profile_hf_generate results in

ImportError: cannot import name 'AttentionType' from 'transformers.models.gpt2.modeling_gpt2'

I find this puzzling since this class is not exported by transformers 4.20.0

Are you using a custom fork of transformers by chance?

We have a speed limit for scraping github, repo homepages at least.
From one ip address it is around 2 repo per second, but it is only 2-3 times faster from 20 different IP addresses ( from the same datacenter, toolkit). A lot of status code 429, rate limiting events. I wonder if it is general github policy or or datacenter just got lucky?
Experiment code here https://github.com/bigcode-project/bigcode-analysis/blob/github_scraping_test/data_analysis/github_scraping_test/github_scrapping_test.ipynb

Maybe anyone can run this experiment on their ray cluster or just repeat the test any other way form their range of ip addresses?

Evaluation datasets like mbpp or HumanEval might be contained in the training set.

• Exact dedup: Remove files containing the exact evaluation code.
• Near dedup: Some of the tasks resemble popular Leetcode questions, how can we identify them at prompt level?

Near deduplication #7 only operates on file level. It is also possible for a file to be

1. a substring of another file, while the minhash/simhash fingerprints being wildly different
2. composed of multiple snippets from different sources

Do we do something about them, knowing they contains large chunks of repeated snippets?