Fork of nanoGPT with updates to train sequence models on midi data.
pip install torch numpy transformers datasets tiktoken wandb tqdm miditok
Dependencies:
- pytorch <3
- numpy <3
transformers
for huggingface transformers <3 (to load GPT-2 checkpoints)datasets
for huggingface datasets <3 (if you want to download + preprocess OpenWebText)tiktoken
for OpenAI's fast BPE code <3wandb
for optional logging <3tqdm
for progress bars <3midtok
for tokenizing midi files
First download the midis zip file from the GiantMIDI-Piano dataset using the link in the disclaimer.md file of the repository. Extract the midi files to the data/giantmidi_piano/midis
folder. Tokenize the dataset by running the tokenize_dataset.py
script from the data/giantmidi_piano
folder. This will create train.bin
and val.bin
files with the tokens.
To start training run
python train.py config/train_giantmidi_piano.py