A sample project for the Harvard ML Foundations group.

Right now this is a demonstration of how to use Singularity containers on the FASRC cluster. An important disclaimer: this is all new to me, so I am confident that some of these "best practices" are in fact suboptimal!

For your own project, you can create a new Singularity definition file based on mlf-sample-project.def, and of course modify the Python requirements listed in requirements.txt.

If the container for your project has already been built by you or someone else, congrats! You can skip to the next section. I've built a container for this sample project, which you can find at /n/holystore01/LABS/barak_lab/Everyone/container_images/mlf-sample-project.sif.

Now, back to business. There are two ways to build a container image from a definition file on the cluster. The first option is to use proot, which must be installed first:

curl -LO https://proot.gitlab.io/proot/bin/proot
chmod +x ./proot

This allows the container to be built without root access. To build an image based on the Singularity definition file, run the following command, replacing /path/to/mlf-sample-project.sif with the path to the image you want to create:

$ singularity build /path/to/mlf-sample-project.sif mlf-sample-project.def

This will take on the order of 10 minutes to complete. Note: singularity is not available from the login nodes, so the above command must be run from a compute or GPU node.

The only downside of the proot approach is that there are a few ways in which it doesn't perfectly emulate root access, so you could hypothetically run into permission problems when building if the definition file calls for certain forbidden actions.

The second option is to create a Sylabs account and build remotely on their free cloud service; instructions here. This may (?) be slower than the proot option; idk, I haven't tried it. Also, the free Sylabs account limits you to performing a total of N builds for some constant N which I forget, and that's not cool.

To enter a shell in the container, run the following command:

singularity shell --nv /path/to/mlf-sample-project.sif

Presto! You now are inside the container and can run commands to your heart's content.

The --nv flag is necessary if you want to use GPUs; otherwise the container's NVIDIA drivers will be tragically unused.

To run a shell command in the container (e.g. in a SLURM batch script), run the following command:

singularity exec --nv --cleanenv /path/to/mlf-sample-project.sif COMMAND

For example, COMMAND might be a Python script.

• The container's file system is read-only. From within the container, you will by default have access to directories on the host's file system
• JupyterLab or code-server will automatically store config files and data in the host system's default locations, so these will not be specific to the individual container. (Storing them inside the container's file system is impossible because this is immutable.)

In the example definition file I have set up a pip virtual environment called venv. I found that if I don't use a virtual environment, Python searches for modules not just inside the container but also on the host system. With my set up, I need to rebuild the container whenever I want to install new Python packages (after adding them to requirements.txt), but I'm willing to accept this inconvenience for the sake of having an environment that isn't contaminated by the host system. (If you don't use a virtual environment, you can install packages on your host system with pip install --user and these will be accessible from within the container.)

This container has JupyterLab installed by default. To run the server, run the following command from within the container:

$ jupyter-lab --no-browser --port=8888 --ip='0.0.0.0'

And then, on your local machine, set up an SSH tunnel:

$ ssh -NL 8888:holygpuXXXXXXX.rc.fas.harvard.edu:8888 [email protected]

with the appropriate values for XXXXXXX and USER.

Code-server is a variant of Visual Studio Code which runs entirely on the remote server — you interact with it through your browser. It is possible to use the standard Visual Studio Code desktop app and ssh into your Singularity containers, but the solution requires your SSH config file to grow as (# cluster nodes) x (# containers), which is a pain. (So far, despite hours of wasted time and misleading conversations with GPT-4, I have failed to find a workable shortcut.)

This container has code-server installed by default. To get the code-server server going, run the following command from within the container:

$ code-server --bind-addr 0.0.0.0:8888

To interact with the server on your local machine, you will need to set up an SSH tunnel locally:

$ ssh -NL 8888:holygpuXXXXXXX.rc.fas.harvard.edu:8888 [email protected]

with the appropriate values for XXXXXXX and USER.

Then, you can access the server at localhost:8888 in your browser. To be clear, the port 8888 is arbitrary; feel free to use your own favorite port (everyone has one).

By default, there is a password stored in ~/.config/code-server/config.yaml. You can remove the password requirement (what could possibly go wrong?...) with the following command:

$ sed -i.bak 's/auth: password/auth: none/' ~/.config/code-server/config.yaml

In Chrome, you can follow these instructions to install the code-server 'progressive web app', which will make it feel more like a native app and make you pine for the real Visual Studio Code less.)

Extensions can be installed as usual; note that they will be installed in your cluster home directory, not in the container's file system, because the container's file system is read-only. Because code-server is based on the open source version of VS Code, some extensions aren't listed by default. In particular, to install Github Copilot (which is the real reason I'm going through all this trouble instead of just using JupyterLab), you will need to download and install it manually:

$ curl -L -o copilot.vsix.gz https://marketplace.visualstudio.com/_apis/public/gallery/publishers/GitHub/vsextensions/copilot/1.89.156/vspackage
$ gunzip copilot.vsix.gz
$ code-server --install-extension copilot.vsix

(To install the current version, go to the VS Code marketplace page and copy the download link.)