johnterickson / cargo2junit Goto Github PK
View Code? Open in Web Editor NEWConverts cargo's json output (from stdin) to JUnit XML (to stdout).
License: MIT License
Converts cargo's json output (from stdin) to JUnit XML (to stdout).
License: MIT License
I'm seeing this error when trying to run my test output through cargo2junit
thread 'main' panicked at 'assertion failed: `(left == right)`
left: `None`,
right: `Some("domains::production_domain_update")`', /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cargo2junit-0.1.13/src/main.rs:211:21
Looks like this is the assertion is the one that is failing.
v0.1.12 seems to have made a breaking change:
$ cargo install cargo2junit
Updating crates.io index
Downloading crates ...
Downloaded cargo2junit v0.1.12
Installing cargo2junit v0.1.12
Downloading crates ...
Downloaded vte_generate_state_changes v0.1.1
Downloaded time-macros v0.2.4
error: failed to compile `cargo2junit v0.1.12`, intermediate artifacts can be found at `/tmp/cargo-install3Oi3Lw`
Caused by:
failed to parse manifest at `/root/.cargo/registry/src/github.com-1ecc6299db9ec823/time-macros-0.2.4/Cargo.toml`
Caused by:
feature `resolver` is required
this Cargo does not support nightly features, but if you
switch to nightly channel you can add
`cargo-features = ["resolver"]` to enable this feature
This isn't how cargo test
behaves. Having no tests is considered a success and it returns 0.
This is an issue for us because we have shared Rust CI code which runs tests, and a small number of our repos have no tests. That works fine for the stage that runs cargo test
today, but if we change that to add ... | cargo2junit
(so test results appear in ADO) then all those repositories have their test stages start failing.
I'm trying to use my little tool as a testbed for using Pipeline Caching for building Rust projects. I hope to standardize a recommendation.
My general approach is:
Some questions?
Here's an example build - just look at VS2019-tar and OSX-tar:
https://dev.azure.com/codesharing-su0/cachesandbox/_build/results?buildId=15929&view=results
and it's YAML:
https://github.com/johnterickson/cargo2junit/blob/cacheTar/azure-pipelines.yml
With Rust/Cargo 1.48 cargo2junit was working fine, but with version 1.50 we are getting this error:
Error: Custom { kind: Other, error: "Error parsing \'{ \"type\": \"test\", \"name\": \"our_crate::tests::test_something\", \"event\": \"ok\", \"exec_time\": 0.000252123 }\': invalid type: floating point `0.000252123`, expected a string" }
It seems that the JSON format has changed between Rust/Cargo 1.48 and 1.50. Previously it looked like this with exec_time
being a string including a unit:
{ "type": "test", "name": "test_something", "event": "ok", "exec_time": "0.005s" }
But with 1.50 it is just a float:
{ "type": "test", "name": "test_something", "event": "ok", "exec_time": 0.001401918 }
Hey, this git repo currently doesn't reflect your releases on crates.io but I think it'd make sense to publish git tags for every crates.io release you make. Could you publish git tags with new crates.io releases from now on?
To me it looks like the output (standard output/error) goes into the wrong attribute/tag (message
).
Taking a look at some documentation: https://llg.cubic.org/docs/junit/ โฆ there seems to be a system-out
, which seems like a better fit.
Writing the output to this tag would improve compatibility with other tooling: https://github.com/inorton/junit2html/blob/248e636278635254d97abcd5d153b08a4254cfbe/junit2htmlreport/templates/report.html#L108-L110
The Rust libs team quietly decided to start blocking -Zunstable-options
in Rust 1.70 outside of nightly builds. Rust 1.70 is now in beta.
$ cargo +beta test -- -Zunstable-options --format json
Finished test [unoptimized + debuginfo] target(s) in 0.22s
Running unittests src/lib.rs ([removed])
error: the option `Z` is only accepted on the nightly compiler
error: test failed, to rerun pass `--lib`
Arguably cargo2unit users were playing with fire by taking a dependency on this... but nonetheless this will strand users on 1.69 who depend on JSON output in their CI builds.
Luckily, it seems that setting RUSTC_BOOTSTRAP=1
in the environment before launching the test will work around this. This seems strictly worse than the status quo since it allows other Rust unstable features to be used as well. Oh well.
Perhaps the README should be updated to reflect this.
I was wondering if it would be possible to break the unit tests into testsuite
based on the module the tests reside in.
Given the following example of a current file:
<?xml version="1.0" encoding="utf-8"?>
<testsuites>
<testsuite id="0" name="darwin" package="darwin" tests="5" errors="0" failures="0" hostname="localhost" timestamp="2022-12-03T21:23:25.627281Z" time="0">
<testcase name="test_one" time="0" classname="cli::test" />
<testcase name="test_four" time="0" classname="data::tests" />
<testcase name="test_two" time="0" classname="data::tests" />
<testcase name="test_three" time="0" classname="cli::tests" />
</testsuite>
</testsuites>
The following would be created
<?xml version="1.0" encoding="utf-8"?>
<testsuites>
<testsuite id="0" name="cli" package="testsuite/cli" tests="5" errors="0" failures="0" hostname="localhost" timestamp="2022-12-03T21:23:25.627281Z" time="0">
<testcase name="test_one" time="0" classname="cli::test" />
<testcase name="test_three" time="0" classname="cli::tests" />
</testsuite>
<testsuite id="1" name="data" package="testsuite/data" tests="5" errors="0" failures="0" hostname="localhost" timestamp="2022-12-03T21:23:25.627281Z" time="0">
<testcase name="test_four" time="0" classname="data::tests" />
<testcase name="test_two" time="0" classname="data::tests" />
</testsuite>
</testsuites>
This would help break things in some CI systems, like Jenkins, into more fragmented chunks; imho.
I know that the CI tools couldn't care less, but it's somewhat inconvenient when printing to stdout, and git will complain if someone wants to check in such a report.
I got the following error on windows:
Error: Custom { kind: Other, error: StringError("Error parsing \'{ \"type\": \"test\", \"event\": \"started\", \"name\": \"src\\lib.rs - (line 17)\" }\': invalid escape at line 1 column 52") }
This might be a serde/json error (seems like it), but it works on the other OSes, so I'm guessing it's OSString or newline related. Maybe it would be possible to normalize the input before JSON parsing it?
OS: Ubuntu 22.04
$ cargo install --locked cargo2junit
Updating crates.io index
Downloading crates ...
Downloaded cargo2junit v0.1.13
Installing cargo2junit v0.1.13
error: failed to select a version for the requirement `time = "=0.3.20"`
candidate versions found which didn't match: 0.3.15, 0.3.14, 0.3.13, ...
location searched: crates.io index
required by package `junit-report v0.7.1`
... which is depended on by `cargo2junit v0.1.13`
Any solutions?
Hadn't seen it on crates.io, but just found https://github.com/mgr-inz-rafal/cargo2junit while searching for my own project. ๐คฆโโ๏ธ
It'd be nice if there were pre-compiled binaries for use in CI pipelines (instead of building it manually).
It would be nice to have the output junit.xml sorted by (pass/fail, time). I typically want to know which ones failed, then which ones were the slowest.
Hi! Thanks for publishing, this package is really useful.
I have one more use case in mind. Is it possible to return non-zero exit code if any test fails?
Example
[user@localhost my-package]$ cargo bench -- -Z unstable-options --format json | cargo2junit > test-results.xml
Finished release [optimized] target(s) in 0.07s
Running target/release/deps/mylib-0841b1c2beb997da
Running target/release/deps/mybin-51a51b3413cebaff
Running target/release/deps/01_bench-0a604b4c894de629
Error: Custom { kind: Other, error: StringError("Error parsing \'{ \"type\": \"bench\", \"name\": \"test_status_code_ok\", \"median\": 1604, \"deviation\": 459 }\': unknown variant `bench`, expected `suite` or `test` at line 1 column 17") }
Could you set the classname attribute only when it is available or maybe to a default value like default?
Having only src/main.rs and src/lib.rs without further modules there is no path:: prefix in the JSON output.
This results in classname="" in the XML output which prevents navigating the test cases in Jenkins.
Recently, Rust has merged a --report-time
flag (rust-lang/rust#64663) into the test library so that the execution time is printed. I think maybe cargo2unit can support this feature.
Hi, I'm sorry if I got something wrong, but in my case I have an fn that sets up tracing subscriber and writes a message to log minimum example:
#[tokio::main]
async fn main() {
println!("Hello, world!");
}
#[cfg(test)]
mod test {
use std::str::FromStr;
use tracing::info;
use tracing_subscriber::fmt::layer;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
use tracing_subscriber::{registry, EnvFilter, Layer};
#[tokio::test]
async fn set_up_subscriber() {
let layer = layer()
.json()
.with_filter(EnvFilter::from_str("info").unwrap_or_default());
registry().with(layer).try_init().unwrap();
info!("Subscriber Initialised");
}
}
unit testing it with:
cargo test | cargo2junit > junit-report.xml
make tests fail:
Error: Custom { kind: Other, error: "Error parsing '{\"timestamp\":\"2022-10-19T06:51:42.193267Z\",\"level\":\"INFO\",\"fields\":{\"message\":\"Subscriber Initialised\"},\"target\":\"test_cargo2junit::test\"}': data did not match any variant of untagged enum Event" }
Any suggestions on how to deal with it ? Thanks in advance.
It would be neat if this tool could also extract the filenames of the thing that's being tested (in the case of integration tests). For example:
Running unittests (target/debug/deps/generate_openapi_spec-e3bc6e08330e8181)
{ "type": "suite", "event": "started", "test_count": 0 }
{ "type": "suite", "event": "ok", "passed": 0, "failed": 0, "allowed_fail": 0, "ignored": 0, "measured": 0, "filtered_out": 0, "exec_time": 0.000019477 }
Running tests/test_api_openapi.rs (target/debug/deps/test_api_openapi-53c722415f8c53df)
{ "type": "suite", "event": "started", "test_count": 1 }
{ "type": "test", "event": "started", "name": "read::can_read" }
{ "type": "test", "name": "read::can_read", "event": "ok" }
{ "type": "suite", "event": "ok", "passed": 1, "failed": 0, "allowed_fail": 0, "ignored": 0, "measured": 0, "filtered_out": 0, "exec_time": 0.06092743 }
As we can see, we can extract the filenames from that text. These would then end up in <testcase file=tests/test_api_openapi.rs/>
which I think would be quite neat. GitLab can actually display those: https://gitlab.com/gitlab-org/gitlab/-/issues/119032
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.