bigcode-project / bigcode-analysis Goto Github PK
View Code? Open in Web Editor NEWRepository for analysis and experiments in the BigCode project.
License: Apache License 2.0
Repository for analysis and experiments in the BigCode project.
License: Apache License 2.0
As we extend deduplication to a wide range of languages, what tokenization method to use will have an impact on the final results.
The current script uses a simple regex and uni-gram to perform minhash calculation. What are the consequences using a different configuration?
Provide results on large dataset with different near deduplication methods:
Details to be included:
Near deduplication #7 only operates on file level. It is also possible for a file to be
Do we do something about them, knowing they contains large chunks of repeated snippets?
The current script building clusters of duplicates, but there are cases it might yield unwanted results:
When doc B is clustered under doc A's name, another doc C can also be clustered into B's name (AB, BC, C!~A), thus when we are deleting non "extreme"s from each cluster, we could end up having both A and B kept in the results.
A better way to delete duplicates is to find community within each connected components. This is used in https://github.com/src-d/gemini.
The link to python-all-license is broken in the data analysis readme.
Loading multi_query_experiments.profile_hf_generate
results in
ImportError: cannot import name 'AttentionType' from 'transformers.models.gpt2.modeling_gpt2'
I find this puzzling since this class is not exported by transformers
4.20.0
Are you using a custom fork of transformers by chance?
We have a speed limit for scraping github, repo homepages at least.
From one ip address it is around 2 repo per second, but it is only 2-3 times faster from 20 different IP addresses ( from the same datacenter, toolkit). A lot of status code 429, rate limiting events. I wonder if it is general github policy or or datacenter just got lucky?
Experiment code here https://github.com/bigcode-project/bigcode-analysis/blob/github_scraping_test/data_analysis/github_scraping_test/github_scrapping_test.ipynb
Maybe anyone can run this experiment on their ray cluster or just repeat the test any other way form their range of ip addresses?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.