Copyright (c) 2020-2021 Yannic Kilcher (yk), Lucas Nestler (clashluke), Shawn Presser (shawwn), Jan (xmaster96)

First, create your VM through google cloud shell with ctpu up --vm-only. This way it has all the necessary permissions to connect to your Buckets and TPUs.
Next, install the requirements with pip on your VM using git clone https://github.com/tensorfork/obst && cd obst && python3 -m pip install -r requirements.txt.
Finally, start a TPU to kick off a training run using python3 main.py --model configs/big_ctx.json --tpu ${YOUR_TPU_NAME}.

• Mesh Tensorflow as machine learning library
• Intial code forked from Eleuther AI's GPT-Neo

We also want to explicitly thank

• tensorfork and TRC for providing us with the required compute (TPUs)
• Ben Wang (kindiana) and Shawn Presser for their invaluable knowledge about TensorFlow, TPU, and language models
• Gwern Branwen, Tri Songz and Aleph Alpha for financing our storage and servers