None of the most important crates outside rustc has a "lib" prefix. If this is supposed to be used by most users, maybe it's a good idea to just call it "test".

I think I've got the right repo but please let me know if I need to move this elsewhere 😄

I was just wondering if you have a ballpark timeline/roadmap for when --format json might be available on Stable?

I'm currently interfacing with the test results output (from an IDE extension) and parsing the pretty format, but having json available on stable would make things a lot easier!

Thanks in advance

My test cases sometimes leave zombie threads hanging around, which sometimes causes issues in unrelated test cases. I'd like to be able to run one test case per process for better isolation, until I can make sure all threads are reaped properly. For the moment I am using the following:

  while IFS= read -r exe; do
    echo "found test exe $exe"
    while IFS= read -r test; do
      echo "found test name $test"
      "$exe" --exact "$test"
    done < <("$exe" --list | grep '.*: test$' | sed 's/: test$//g')
  done < <(cat cargo-output.json | jq -cMr 'select(.reason == "compiler-artifact" and .target.kind == [ "test" ]) | .executable')

where cargo-output.json is cargo's message-format output. This is far from ideal, and I'd love support for this in libtest.

Do you reckon there are technical reasons why it can't be made?

2 years without changes and the main rust repo seems to have a very similar looking https://github.com/rust-lang/rust/tree/master/library/test/src

If this repo is not used anymore, could that please be made more obvious?

I'd like to be able to get a json-formatted list of the tests and benches. This isn't a huge deal, because the output is pretty simple to parse with regex.

Expected:

> test_binary -Z unstable-options --list --format json
[{name: "foo", type: "test"}]

Actual:

> test_binary -Z unstable-options --list --format json
foo: test

1 test, 0 benchmarks

I'm using a nightly compiler (by running rustup install nightly && rustup default nightly) and ran the following testing command,

cargo test -- --include-ignored

This command yields the error

error: The "include-ignored" flag is only accepted on the nightly compiler

I have also tried adding the +nightly option, that didn't work either.

The version of the compiler I'm using is

nightly-x86_64-unknown-linux-gnu unchanged - rustc 1.39.0-nightly (6ef275e6c 2019-09-24)

We should come up with a couple of goals for the roadmap of what to do next. The first goal is probably making the crate usable. For that we need to:

• get it running on stable Rust, and
• get rust-lang/rust to use it as is.

To get it running on stable Rust we can just try to stop using unstable Rust features. To get rust-lang/rust to use it, the main blocker was the relationship between #[ignore] and #[test] which was discussed in the internal thread (https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139).

@alexcrichton proposed the following:

I’m not sure I completely follow the trickiness with #[test] and #[ignore] orderings, but it seems to me like we could make both of them procedural macros in the prelude by default perhaps? If both #[test] and #[ignore] were routed to the same procedural macro, it wouldn’t matter which runs first and #[ignore] would just otherwise verify that #[test] exists (removing it).

In the meantime there are also a couple of things that we might want to do, but that's probably better to discuss them in their own issues:

• refactor argument parsing, formatting, json, etc. to make them reusable for other custom test frameworks
• refactor testing and benchmarking into different crates, so that one doesn't need to pull in the whole testing infrastructure for benchmarks and vice-versa
• use the real term crate instead of our own fork
• make sure that libtest is usable as a custom_test_framework (and potentially "libbench" as well if we have splitted them in two)

cc @djrenren

We need to set up CI for the tier-1 and tier-2 targets. We probably want to do something similar to what libc, stdsimd, etc do.