Note that gvm will set your GOPATH and PATH variables properly to allow you to run Go binaries that you install. However, if you are interested in writing Go code you should set a more accessible GOPATH as described as described here.

In order to submit to the test161 server, you need to configure your username/token, which can be done with:

Removing users and changing configured tokens can be done with:

The ops-class sources create symlinks between your OS/161 source directory and root directory, which may not always be what you want. To support partial environments where either source or root cannot be inferred from the other, you can use test161 config to set the directory containing tests, targets, and commands:

This allows you to run test161 run …​ from your root directory.

To run a tests, group of tests, or target, use the test161 run <names> sub-command. Here, names can be a single target, a list of tests, or a list of [tags].^[1] For test files, <names> is a list of http://www.linuxjournal.com/content/globstar-new-bash-globbing-option [globstar] style file names. The following are all valid commands:

conf:
  cpus: 8
  ram: 1M
  disk1: # disabled by default, but should be enabled when you want swap disk
    sectors: min(2 * ram, 8000) # 8000 is a minimum due to a current sys161 bug
    rpm: 7200
    nodoom: true
  disk2: # disabled by default, but uses these defaults if configured
    sectors: min(2 * ram, 8000)
    rpm: 7200
    nodoom: false
  random: seed=random # random number generated at configuration time
stat:
  resolution: 0.01
  window: 1
monitor:
  enabled: "true
  intervals: 10
  kernel:
    enabled: false
    min: 0.001
    max: 1.0
  user:
    enabled: false
    min: 0.0001
    max: 1.0
  timeouts:
    prompt: 300
    progress: 10

The basic unit in test161 is a command. such as lt1 for running Lock Test 1, or sp1 to run the whalemating test. Information about what to expect when running these commands, as well as what input/ouput they expect is specified in the commands/ directory in your test161 root directory. All .tc (test command) file in this directory will be loaded and commands must only be specified once.

Test files (*.t) are located in the tests/ directory in your test161 root directory. This directory can contain subdirectories to help organize tests. Each test consists of one or more commands, and each test can have its own sys161 configuration. Tests are run in their own sandboxed envrionment, but commands within the test are executed within the same sys161 session. Some tests will consist of multiple commands, and such tests are designed to stress test your system.

Target files (*.tt) are located in the targets/ directory in your test161 root directory. Targets specify which tests are run for each assignment, and how the scoring is distributed. When you test161 submit your assignments, you will specify which target to submit to.

A line starting with $ will be run in the shell and start the shell as needed. Lines not starting with $ are run from the kernel prompt and get there if necessary by exiting the shell. sys161 shuts down cleanly without requiring the test manually exit the shell and kernel, as needed.

So this test:

Expands to:

Note that commands run in the shell must be prefixed with $. Otherwise test161 will consider them a kernel command and exit the shell before running them. For example:

This test is probably not what you want:

Because it will expand to:

But this is so much simpler, right?

Optionally, tests can have one or more tags. test161 can be invoked to run these tests as a group with test161 run <tag>.

test161 uses the collected stat161 output produced by the running kernel to detect deadlocks, livelocks, and other forms of stalls. We do this using several different strategies:

1. Progress and prompt timeouts. Testfiles can configure both progress (monitorconf.timeouts.progress) and prompt (monitorconf.timeouts.prompt) timeouts. The former is used to kill the test if no output has appeared, while the latter is passed to expect and used to kill the test of the prompt is delayed. Ideally OS/161 tests should produce some output while they run to help keep the progress timeout from firing, but the other progress tracking strategies described below should also help.

2. User and kernel maximum and minimum cycles. test161 maintains a buffer of statistics over a configurable number of stat161 intervals. Limits on the minimum and maximum number of kernel and user cycles (expressed as fractions) over this buffer can help detect deadlocks (minimum) and livelocks (maximum). User limits are only applied when running in userspace.

3. Note that test161 also checks to ensure that there are no user cycles generated when we are running in kernel mode, which could be caused by a hung progress.

test161 supports different output strategies through its PersistenceManager interface. Each TestEnvironment as a PersistenceManager which receives callbacks when events happen, like when scores changes, status change, or when output lines are added. This allows multiple implementations to handle output as they wish. The test161 client utility implements the interface through its ConsolePersistence type, which writes all input to stdout. The server uses a MongoPersistence type which outputs JSON data to our mongo backend server.

• Colored test output on terminals that support it? (Particularly for correct/incorrect.)

• Handle missing newline correctly. Test with shll for lossy shell support.

• sys161 version checks

• Order the test output in some meaningful way, probably by depth in the dependency graph. (That way all skipped tests should be shown last.)

  • Not necessarily true. You could have a long, unrelated branch that succeeds (even leaf nodes), but some unrelated depencency fails early. I made the default print order topological sort, but it’s still confusing. I added the reason a test is skipped, which helps. Maybe a nice ASCII art tree would work here…​

• Check for repository problems:

  • Check and fail if it has inappropriate files (.o), or is too large. (Prevent backend storage DOS attacks.)

• Use URL associated with the tree-ish id provided to test161 submit

• Fix directory bash completion for test161 config test161dir. It’s unfortunately adding a space instead of /.

Most of the infrastructure is in place to handle performance targets, but we still need finish this and test it. Specifically, we need set the performance number in the Test object and use it properly in the Submission.

It would be cool to be able to print serial output from one test while queuing the output from other tests. Maybe using curses to maintain a line at the end of the screen showing the tests that are being run.

It would be nice to be able to add descriptions to tags and have test161 print all tests that fall into a tag (or target) along with the description.

For long running tests, OS/161 tests generate periodic output, usually in the form of a string of '.' characters. This output is used as a keep-alive mechanism, resetting test161’s progress timeout. Because this output is in a single line, and it would create more unnecessary DB output and server load to break these into multiple lines, it would be nice to refactor things in such a way that the current output line is periodically persisted. This would give students a better indication of progress, as opposed to tests looking "stuck".

Now that we are having students save a key through the web interface we need to make sure that these keys get saved, associated with each successful submission, and not destroyed even if they are changed later.

It would be great if test161 run boot.t --sys161-cpus=1 worked properly. I think that there is a library for this.

It should be possible to automate the process of hooking up a debugger and running BT on panics.

• Environment inference with environment variable overrides, similar to the test161 client

  • test161-server config to both show the configuration and modify it

• Log the configuration on startup

• More usablility cleanup

  • Usage

  • Help

  • Bash completion

• Moving window for stats API

• Periodically persist server stats, either in MongoDB or through the logger. We currently lose these on restart.

• Move collaboration messages into their own files instead of hard-coding