itsuncheng / fine-tuning-gpt2 Goto Github PK
View Code? Open in Web Editor NEWCodebase for the Medium Article on Fine-tuning GPT2 for Text Generation
Codebase for the Medium Article on Fine-tuning GPT2 for Text Generation
Hi there, I recently found this repository and your post explaining how everything works and I have to say its pretty good but on your post you mention that we can continue training from the last checkpoint but it only works if first you run it until it reaches the first training epoch or if it finishes processing everything. I tried running the code on a huge dataset with billions of parameters and I could not stop it and resume it because it would take too long before it reached the first training epoch, once I stopped it I couldn't resume it as there were some files that were not created by the run_language_modeling.py
script, I also tested the code with a smaller dataset and there I was able to resume it as the tokenizer_config.json
, vocab.json
and other files needed to resume the training were created after it ended. I think it would be a lot more useful if all those files needed to resume the training were created into each checkpoint folder, I know it will probably make the checkpoint use more space but that's should not be a problem as it won't be that much of extra space we will be using.
Thx for your implementation. It's kinda annoying that the model saves checkpoint every 500 steps when training on large dataset.
This is only support of python2, right? If I want to use python3, how do it? Many thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.