Should be possible if we grab the doctest executables through -Z unstable-options --persist-doctests.

This needs to be done with a little care:

• The initial, simpler use requires keeping track of which Cargo arguments the build was run with -- this is relatively simple.
• An extended use case to solve is "what if the user wants to grow the set of tests or binaries that are run?" This is going to require us keeping track of the exact set of binaries run and Cargo arguments passed. Probably worth discussing with some folks before doing so.

Overall this is a stateful operation, kind of like a source control bisect.

"classname" seems to be a unique identifier for the immediate container for each test. We should be able to construct it with package-name.[binary-name.]module.path, using . rather than ::.

Add support to run a single test multiple times in parallel. This is a somewhat different mode of operation from nextest as usual (the runner should be fed the same test multiple times, and should maybe not run other tests at the same time), but is worth doing as a future improvement.

It is possible to write a test that ends up leaking a pipe (e.g. creates a process but doesn't terminate it). Currently, the test runner hangs on encountering such a test.

Instead, we should figure out a way to:

• detect such a situation (a small amount of raciness between waiting on process exit and checking that the handles are closed is fine)
• mark such a test as failure with a LEAK message or similar

At the moment we aren't publishing any timestamps, just test durations. It would be great to have support for timestamps.

One issue is that the JUnit format doesn't support timestamp in testcase elements, only in testsuite elements. This means that we'll probably have to wrap each testcase in its own testsuite. Note that testsuite elements can be nested.

Add a mechanism to the test runner to capture system CPU/mem/disk utilization and upload it with the other test runner results.

Probably can be done with a background thread.

TODO: think about the data model

cc @rexhoffman

Make the datatest runner more deterministic.

Developer reported a unit test job (https://github.com/diem/diem/actions/runs/836112503) that hung for over 30 minutes without output. In this case, test runner should log when a test is running longer than a preset time limit.

2021-05-12T16:31:34.1985590Z         PASS [   1.897s]                                             vm-validator  vm_validator::vm_validator_test::test_validate_module_publishing
2021-05-12T16:31:34.2755302Z         PASS [   1.876s]                                             vm-validator  vm_validator::vm_validator_test::test_validate_module_publishing_non_association
2021-05-12T16:31:34.3417959Z         PASS [   1.863s]                                             vm-validator  vm_validator::vm_validator_test::test_validate_non_genesis_write_set
2021-05-12T16:31:34.3506958Z         PASS [   1.872s]                                             vm-validator  vm_validator::vm_validator_test::test_validate_sequence_number_too_new
2021-05-12T16:31:34.4389504Z         PASS [   1.918s]                                             vm-validator  vm_validator::vm_validator_test::test_validate_transaction
2021-05-12T16:31:34.6665912Z         PASS [   1.811s]                                             vm-validator  vm_validator::vm_validator_test::test_validate_unknown_script
2021-05-12T16:31:39.9518323Z         PASS [  11.483s]                                  ir-testsuite::testsuite  run_test::move/sorted_linked_list/sorted-linked-list.mvir
2021-05-12T17:01:58.1447546Z ##[error]The operation was canceled.
2021-05-12T17:01:58.1529807Z ##[group]Run actions/upload-artifact@v2
2021-05-12T17:01:58.1530372Z with:
2021-05-12T17:01:58.1530832Z   name: unit-test-results
2021-05-12T17:01:58.1531442Z   path: target/junit-reports/unit-test.xml
2021-05-12T17:01:58.1532057Z   if-no-files-found: warn
2021-05-12T17:01:58.1532502Z env:
2021-05-12T17:01:58.1532873Z   max_threads: 16

Right now the only communication channel we have between the testrunner and Rust tests, during execution, is the process exit code. We'd like to have a richer protocol but the Rust test harness is kind of limited so we need to be careful about it. Some ideas for things to communicate:

• test flakiness
• test retries
• test timeouts

Some ideas for the protocol itself:

• Use lines to stdout (or stderr) that begin with nextest: (similar to cargo: lines in build scripts)
• Pass in a random string through the environment and ensure that lines begin with that to avoid confusing random test output with nextest control lines.
• Publish a crate which can be used to output these lines, either as function calls or as a proc macro.

The default for cargo test is to quit on the first failing test (well, the first failing binary). The test runner should have a way to do that as well.