Fork of nanoGPT with updates to train sequence models on midi data.

pip install torch numpy transformers datasets tiktoken wandb tqdm miditok

Dependencies:

• pytorch <3
• numpy <3
• transformers for huggingface transformers <3 (to load GPT-2 checkpoints)
• datasets for huggingface datasets <3 (if you want to download + preprocess OpenWebText)
• tiktoken for OpenAI's fast BPE code <3
• wandb for optional logging <3
• tqdm for progress bars <3
• midtok for tokenizing midi files

First download the midis zip file from the GiantMIDI-Piano dataset using the link in the disclaimer.md file of the repository. Extract the midi files to the data/giantmidi_piano/midis folder. Tokenize the dataset by running the tokenize_dataset.py script from the data/giantmidi_piano folder. This will create train.bin and val.bin files with the tokens.

To start training run

python train.py config/train_giantmidi_piano.py