My submission to the Nanda stream of the 2024 MATS program - ten hours of work on a concrete subproblem within mechanistic interpretability.
Google doc explaining the results is here.
Colab link is here.
Complication to the time requirement is here:
POV: you have realised that your ancient desktop with a fancy new 4070 SUPER no longer has a functional Linux install, but the deadline for the MATS application is in 2 days.
- Install VSCode.
- Install WSL.
- Realise that you don't have virtualization enabled in your BIOS - go to 'Advanced Frequency -> Enable SVM', since its AMD.
- Install Windows Terminal, boot Ubuntu (
wsl
). - Generate new SSH keys, add to Github.
- git clone your new repo from your WSL terminal, use
code .
to open the repo in WSL/VSCode. - From then on, use the integrated terminal. Use
eval "$(ssh-agent -s)"
thenssh-add
to ensure you can push commits. - Install zsh/oh-my-zsh (useful for gco, gst aliases).
- Install conda, create an mechinterp env. (
conda create -n mechinterp python=3.11 numpy scipy matplotlib scikit-learn
) - Install the Jupyter and Python extensions for VSCode.
- Activate your conda env, install Jupyter.
- Install torch with CUDA (surprisingly easy -
conda install pytorch pytorch-cuda=11.8 -c pytorch -c nvidia
)