itsuncheng / fine-tuning-gpt2 Goto Github PK

Hi there, I recently found this repository and your post explaining how everything works and I have to say its pretty good but on your post you mention that we can continue training from the last checkpoint but it only works if first you run it until it reaches the first training epoch or if it finishes processing everything. I tried running the code on a huge dataset with billions of parameters and I could not stop it and resume it because it would take too long before it reached the first training epoch, once I stopped it I couldn't resume it as there were some files that were not created by the run_language_modeling.py script, I also tested the code with a smaller dataset and there I was able to resume it as the tokenizer_config.json, vocab.json and other files needed to resume the training were created after it ended. I think it would be a lot more useful if all those files needed to resume the training were created into each checkpoint folder, I know it will probably make the checkpoint use more space but that's should not be a problem as it won't be that much of extra space we will be using.

Thx for your implementation. It's kinda annoying that the model saves checkpoint every 500 steps when training on large dataset.

This is only support of python2, right? If I want to use python3, how do it? Many thanks.