Run untrusted code in an isolated environment
Sandkasten is a code execution engine for running arbitrary untrusted/harmful code in a sandbox, isolating it from both the host system and other Sandkasten jobs. A simple REST API allows uploading and executing arbitrary programs, while also enabling the user to specify resource limits and providing feedback on the actual resources used. This project was partly inspired by Piston and aims to solve some problems with it.
Sandkasten uses nsjail to run programs in restricted environments and to enforce the specified resource limits. Additionally GNU Time is used for reporting the resources used by the program. Programs are always run in a chroot environment using nsjail, which contains only the following directories:
/program
(rw in compile steps, ro in run steps) contains the compiled program/box
(ro) current working directory which contains the specified files for compile/run steps/tmp
(rw, tmpfs)- the paths in
/nix/store
that are needed by the selected environment (ro mount from host) - some files in
/dev
and/etc
which are needed for some packages to work properly
Programs are uniquely identified using the hash value of their source files and selected environments. If a program has been uploaded and compiled before and is then uploaded again, the same program id is used and the existing compilation results can be used without having to recompile the program.
- Compile and execute arbitrary programs.
- Cache compilation results to avoid having to recompile the same programs for every time they are run.
- Set resource limits for both compile and run steps.
- Report resource usage for both compile and run steps.
- Packages are defined using Nix.
- Programs are deleted automatically if they are not executed anymore.
- Specify stdin, command line arguments and files in the working directory for run steps.
- Specify environment variables for both compile and run steps.
- Client library for Rust (crate, documentation)
- Communicate with running programs via websockets.
- Spawn multiple processes that can communicate with each other.
- JWTs for individual limits (+rate limits).
- Add more packages.
On a running Sandkasten instance, the API documentation is available on <instance>/docs
and
<instance>/redoc
. There is also an OpenAPI specification available on <instance>/openapi.json
.
A public test instance is available at https://sandkasten.bootstrap.academy/. Please note that there is a rate limit of 20 requests per minute (if you exceed this limit, you may receive 429 errors). Also, this instance is not intended for production use, it is currently recommended to host your own instance for that (see instructions below).
The recommended way of installing Sandkasten is to setup a dedicated virtual machine running NixOS. To make this setup easier, this repository contains a basic NixOS configuration template and an installation script.
The following steps have been tested on Proxmox VE 7.4-3 x86_64, Proxmox VE 8.0.3 x86_64 and libvirtd 9.4.0 x86_64.
- Download the minimal NixOS ISO image from https://nixos.org/download.html#nixos-iso
- Create a new virtual machine.
- Disk size: at least 16GB
- Internet connection and DHCP server to get an IPv4 address are required.
- Start the virtual machine and boot into the NixOS installer.
- Run
sudo su
to obtain root privileges. - If necessary, change the keyboard layout (e.g.
loadkeys de
for german qwertz layout). - Use
lsblk
orfdisk -l
to find the name of your hard disk. - Run the following commands to download the installation script from GitHub and start the
installation. Replace
[disk]
with the path to your hard disk (e.g./dev/sda
). Note that this will erase all data on the disk you specify.curl -o install.sh https://raw.githubusercontent.com/Defelo/sandkasten/latest/install-vm.sh bash install.sh [disk]
- After the script is done, the vm will reboot into the new NixOS installation. The initial root
password is
sandkasten
if you want to login via ssh. The Sandkasten server is started automatically and should be listening on0.0.0.0:80
by default.
In /root/sandkasten
you can find a flake which contains the
configuration of your vm. To update Sandkasten and the system packages run
nix flake update && nixos-rebuild switch --flake .
in this directory. Here you will also find
a sandkasten.nix
which contains the Sandkasten service configuration, a configuration.nix
which contains the system configuration and a hardware-configuration.nix
which is generated
automatically by the install script. After making changes to any of these files run
nixos-rebuild switch --flake .
to apply them.
Follow these steps if you want to install Sandkasten on an existing (flakes based) NixOS installation:
- Add this repository to your flake inputs:
{ inputs.sandkasten.url = "github:Defelo/sandkasten/latest"; }
- Add the module to your NixOS configuration:
{ imports = [sandkasten.nixosModules.sandkasten]; }
- Configure the module:
{ services.sandkasten = { enable = true; environments = p: with p; [ rust python typescript # use `all` to install all environments ]; # example config: host = "0.0.0.0"; port = 8080; max_concurrent_jobs = 16; run_limits.time = 10; # for a full list of configuration options, see `config.toml` }; }
The following components are needed for a working development environment:
If you have direnv installed, you can just use
direnv allow
to setup your shell for development. Otherwise you can also use nix develop
to enter a development shell. This will add some tools to your PATH
and set a few environment
variables that are needed by Sandkasten and some of the integration tests.
The first time you enter
the development shell, you should run the setup-nsjail
command, which will copy the nsjail
binary into your current working directoy, chown
it to root
and set the setuid
bit to allow
Sandkasten to run this binary as root without having to run Sandkasten itself as root (but of
course you could also do that).
Before starting Sandkasten, you should setup a Nix profile with the environments that you want to be available on your instance. A full list of installable environments is available at nix/packages. To install a package, you can use the following command:
nix profile install --profile pkgs .#packages.<package-name>
If you want to install all packages, use all
for <package-name>
. You can also add, upgrade or
remove packages later, but you need to restart Sandkasten after doing so. See
nix profile --help
for details.
Sandkasten uses Redis for caching. To start a redis server for development,
simply run redis-server
in the development shell. Otherwise, if you already have a redis server
running somewhere, that you would like to use instead, set the environment variable REDIS_URL
to the url of your redis server (see https://docs.rs/redis/latest/redis/#connection-parameters for
details).
In the development shell you can just use cargo run
to start Sandkasten.
To run the unit tests, you can just use cargo test
. This only requires you to have a working rust
toolchain, but you should not need to setup nix for this.
To run the integration tests, you can use cargo test -F nix -- --ignored
. For this to work you
need to have a Sandkasten instance running on 127.0.0.1:8000
. You can also specify a different
instance via the TARGET_HOST
environment variable. If you only want to run the integration tests
that do not require a nix development shell, you can omit the -F nix
. In the development shell you
can also run the integration-tests
command to automatically start a temporary sandkasten instance
and run the integration tests against it. There is also a cov
command that runs the integration
tests and writes an html coverage report to lcov_html/index.html
.
All packages are defined using nix expressions in nix/packages. Each package has a unique id, a human-readable name, a version, optionally a script to compile a program, a script to run a program and a test program that is executed as part of the integration tests to ensure that the package is working.
The compile script of a package is executed whenever a new program has been uploaded. When this
script is run, the current working directory (/box
) contains all the source files and the command
line arguments contain the names of the source files in the same order as they were specified by
the client (starting with main_file
which represents the entrypoint into the program). The purpose
of the compile script is to compile the provided program and store the result (plus any files that
may be needed to run the program) in /program
.
If a package does not have a compile script, the source files are instead copied directly into the program directory.
The run script of a package is executed whenever a program is executed. When this script is run, the
current working directory (/box
) contains the files that have been specified in the run step (if
any) and /program
contains the files that have been produced by the corresponding compile script
previously (or the source files if the packages does not have a compile script). The first command
line argument is always the name of the main_file
(which represents the entrypoint into the
program). In most cases, this is only relevant for interpreted languages (like Python) and can be
ignored for most compiled languages. All other command line arguments are the ones specified by the
client and should be forwarded to the actual program.
Every package should provide a test program that checks the following:
- Multiple source files are working (e.g.
first_file.py
can importsecond_file.py
) - Reading from stdin is working. The program should assert that the string
stdin
is read from stdin. - Command line arguments are working. The program should assert that the only three command line
arguments are
foo
,bar
andbaz
. - File system is working. The program should assert that the file
test.txt
in the current working directory contains the stringhello world
.
If any of these checks fails, the program should exit with a non-zero exit code. Otherwise, if all
checks passed, it should exit with exit code zero and print OK
to stdout.