Looking at https://docs.rs/assert_cmd/0.9.1/assert_cmd/, what should be imported to top-level and what shouldn't?

Examples

• It'd be nice if the docs listed all top-level stuff together.
• assert, cargo contains details that aren't as relevant for most users and shouldn't be top-level

I want compare command stdout with my tested app's stdout like this:

    let output = Command::new("uname").output().unwrap();
    let stdout = output.stdout;
    new_cmd!()
        .assert()
        .success()
        .stdout(stdout)    // this will give an error
        .stderr("");

I can't find a simple API from predicates. Can you suggest a proper way to handle this?

#57 / #4 is about a performance vs correctness issue in assert_cmd. We should document the workaround for people who want the performance at the cost of correctness (or rather, they can ensure the correctness).

In how we document this: I noticed #6's documented workaround is in the cargo module. The challenge is if people look at the trait or functions directly. We should find a way to address that for that workaround and this one.

Hi, maybe is a my mistake, but i cant use the String type with 'stdout', only the str type.

    let mut main = Command::main_binary().unwrap();

    main.arg("tags")
        .arg("list")
        .arg("--use-csv");

    let stdout = String::from("\"Name\",\"#id\"");

    // stdout.push_str ....
    // stdout.push_str ....
    // stdout.push_str ....
  
    main.assert()
        .success()
        .stdout(stdout.as_str());

On run this, i receive error[E0597]: stdout does not live long enough. I cant use .stdout(stdout.as_str());, because i receive the trait assert_cmd::assert::IntoOutputPredicate<_> is not implemented for std::string::String.

Where am I going wrong :( ?

Thank you in advance for your attention.

In getting feedback, there came up the need for timeout support.

We'd have to run the command in a thread, sleep for the time (in small increments?), and then call kill if needed.

https://doc.rust-lang.org/std/process/struct.Child.html#method.kill

Examples

• In Assert, OutputError, OutputResult, and StdinCommand, how to construct them
• Update one-line summaries to clarify the role (factory, etc)

For #24 we added support for passing a file's contents to stdin (see #26).

Should we make it stream contents in?

• From a file, read and pass in a chunk at a time
• Support a Read object being passed in

Hi. I'm currently a user of assert_cli in tealdeer and am thinking about moving to assert_cmd. The main issue I'm facing there is that the binary (when calling Assert::main_binary()) is always compiled with default features, even if I compile the tests with --no-default-features.

Does assert_cmd have a solution for this? Ideally I'd either be able to pass the features to the main_binary or cargo_bin methods, or would have a way to specify the features per-invocation by other means (e.g. set_features(&["networking"]) or set_features(&[])). This way, I could test different feature flags from a single test binary.

Alternatively compiling the main binary with the same flags as the test binary would be fine too.

The with_stdin will take the ownership of command. Therefore you cannot write chained functions like this:

Command::new("cat")
    .arg("-A")
    .with_stdin("42")
    .assert()
    .success();

Error:

error[E0507]: cannot move out of borrowed content
  --> /xxx.rs:43:5
   |
43 | /     Command::new("cat")
44 | |         .arg("-A")
   | |__________________^ cannot move out of borrowed content

From @luser on #28

You actually might want to consider refactoring the current code to write stdin on a separate thread and then read stdout/stderr in parallel to avoid possible deadlock. Right now if the spawned process produces enough output to fill up its stdout or stderr buffers it'll block trying to write those, and if you are still writing stdin data you'll be blocked as well. I've got a wait_with_input_output function that does this in sccache which you can copy if you'd like. (Linking to an older version because the current source uses tokio-io and futures.)

So far, we haven't run into this but

I suspect when running cargo test --target TRIPLET that main_binary will still use the default TRIPLE.

Downside: when the user is calling cargo test without a --target, assert_cmd passing --target will cause a one-time large compile time cost

See assert-rs/assert_cli#109

As discussed in this PR mesalock-linux/mesabox#24. We should use predicate::str::contains(str).from_utf8() to convert Predicate<str> to Predicate<[u8]> for stdout() and stderr() comparison.

It's difficult to find answer in the documentation. There are several reasons:

1. assert_cmd and predicate's documentation are separated, you should look it back and forth.
2. The from_utf8() is in the PredicateStrExt trait, and it's difficult to find.

For example, my intuition is that stdout() accept a str like this: stdout("Hello, World!"). But what if I want to check if it contains this str? Then I found that stdout() accept a IntoOutputPredicate<P> actually. Then I switch to read the doc of IntoOutputPredicate and understand str implement this trait. But still, how to check contains? I read the doc of predicate and find out str::contains. The doc doesn't tell me how to use it in assert_cmd, then I'm confused. contains() gives me a ContainsPredicate. However, ContainsPredicate only implements Predicate<str>. At this point, I begin to be impatient and spent a lot of time reading the doc. Finally I find the PredicateStrExt.

This is my first experience of using assert_cmd and predicate. I don't know how to improve the APIs. I think it worth to improve the docs, like giving more examples combining assert_cmd and predicate.

Hiya! I've appeared here since assert_cli is deprecated, and tried swapping over from assert_cli 0.6 to assert_cmd 0.9

My really short experience (so far) has been something like this:

1. Let's swap over the assert_cli to assert_cmd in the imports
2. Okay, so Commands from std are extended by traits, cool!
3. How do I get environment stuff in? It's not mentioned in these docs, let me check std, okay it's there.
4. How do I write things to stdin? Searched and found CommandStdInExt.
5. What's .with_stdin().buffer("..").unwrap()? Why can't I call .buffer("") twice? Oh we're no longer the builder.
6. Is this unwrap() the assertion? No wait it just gives me the StdinCommand
7. How do I assert? Searched and found OutputAssertExt.
8. The example on the output extension page is a trivial example. How did it get an Output again? Oh it's a std type.
9. Okay I had a StdinCommand, or was it a Result<StdinCommand>.
10. Oh man. I have successfully been confused.
11. Let's open many tabs.
12. No wait let's open an issue.

There's a lot of jumping around trying to figure out "what type do I have, is it a Builder, a Result, a normal type by extended by some trait". Also, the Result's errors aren't all the same type (and aren't convertible), so I can't easily go main_binary()? and then later on .output()?.assert().success().

I think we need an example on the first page (and perhaps README.md) that does a complete test. The piecewise explanations are good on their own, but it's hard to tie them together. That is, literally every method call gives me a different type:

extern crate assert_cmd;

use assert_cmd::{cargo::CargoError, prelude::*};
use std::process::Command;

#[test]
fn integration_test() -> Result<(), CargoError> {
    Command::main_binary()?
        .env("APP_VAR", "value") // https://doc.rust-lang.org/std/process/struct.Command.html#method.env
        .with_stdin()            // https://docs.rs/assert_cmd/0.9.1/assert_cmd/trait.CommandStdInExt.html
        .buffer("exit\n")        // https://docs.rs/assert_cmd/0.9.1/assert_cmd/struct.StdInCommandBuilder.html#method.buffer
        .output()                // https://docs.rs/assert_cmd/0.9.1/assert_cmd/struct.StdInCommand.html#method.output
        .unwrap()                // Gives you back the `Output`
        .assert()                // https://docs.rs/assert_cmd/0.9.1/assert_cmd/assert/trait.OutputAssertExt.html#tymethod.assert
        .success();              // https://docs.rs/assert_cmd/0.9.1/assert_cmd/assert/struct.Assert.html#method.success
    Ok(())
}

FWIW I was trying to port this test over:

let environment =
    assert_cli::Environment::inherit().insert("APP_DIR", env!("CARGO_MANIFEST_DIR"));

assert_cli::Assert::main_binary()
    .with_env(&environment)
    .stdin("exit\n")
    .unwrap();

_{Off topic, CONTRIBUTING.md looks like it might have been copied from something called stager.}

From @rcoh assert-rs/assert_cli#100

• assert_cli version: 0.5
• Rust version: 1.25.0
• OS and version: Ubuntu 16.04
  Unfortunately, cargo run takes at minimum about 300ms on my computer:

➜  angle-grinder git:(master) ✗ time cargo run --bin agrind -- '* | json | count' --file test_files/empty
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/agrind '* | json | count' --file test_files/empty`
No data
cargo run --bin agrind -- '* | json | count' --file test_files/empty  0.22s user 0.06s system 92% cpu 0.307 total

It makes it impractical to run hundreds of integration tests with assert_cli.

I'm not positive what a better way would be -- I guess running the binary in dist directly if it exists?

Workaround

See assert_cmd::cargo's docs

As described mesalock-linux/mesabox#41. I don't know why the first tests are always blocking for over 60 seconds. However, after the blocking issue, all testcases can be run without any problem.

test gnu::arch::test_x86_64 ... test gnu::arch::test_x86_64 has been running for over 60 seconds
test gnu::base32::test_decode ... test gnu::base32::test_decode has been running for over 60 seconds

This only happens in Travis: https://travis-ci.org/mesalock-linux/mesabox/jobs/411140601#L1305

I cannot reproduce this issue in my machine. Don't know why.

For example, I want to assert a command which only have specific stdout but does not have any stderr message. Currently, I have to write .stdout("msg").stderr("") or stdout("").stderr("error msg").

The API was thrown together using failure::Error but our library should provide a concrete error that implements Fail instead.

Hi! Today, I've written a bunch of tests for https://github.com/ferrous-systems/cargo-review-deps/blob/3ba1523f3b2c2cfe807bc76f42bf972d0efdc113/tests/test_cli.rs.

Initially, I've started with assert_cmd, then I've switched to assert_cli and now I have "write my own minimal testing harness on top of std::process::Command" on the todo list. I'd like to document my reasoning behind these decisions. Note that these are just my personal preferences though!

The first (but very minor) issue with assert_cmd was it's prelude/extension traits based design. It makes for an API which reads well, but it is definitely a speed bump for the new users who want to understand what API surface is available to them. I think that long term such design might be an overall win, but it is a slight disadvantage when you learn the library.

The issue which fundamentally made me think "I'll stick with assert_cli for now" was the predicate-based design of assertions for stdout. It has two things which don't fit my style of testing. First, the same issue with prelude. The first example for .stdout starts with

extern crate assert_cmd;
extern crate predicates;

use assert_cmd::prelude::*;

use std::process::Command;
use predicates::prelude::*;

That's an uncomfortable for me amount of things I need to import to do "quick & dirty smoke testing".

The second thing is extremely personal: I just hate assertion libraries with a passion :) Especially fluent assertion ones :-) I've tried them many times (b/c they are extremely popular), but every time the conclusion was that they make me less productive.

When I write tests, I usually follow the following stages.

• A simple test with assert!(a == b), without custom message. This the lowest-friction thing you can write, and making adding tests easy is super important.
• The second stage happens when the test eventually fails. I see assert!(false) in the console, and go to the test and write an elaborate hand-crafted error message with all kind of useful information, to help me debug the test.
• The third stage (which I call data-driven testing) happens when I have a bunch of tests with similar setups/asserts. What I do then is that I remove all the asserts from all the tests by introducing a special function fn do_test(input: Input, expected: Expected) which turns arrange and assert phases of a test into Data (some JSON-like pods, a builder DSL or just a string with some regex-based DSL). Internally, do_test does all kind of fancy validation of input and custom comparisons of expected and actual outcomes.

I feel that assertion libraries sit somewhere between 1 and 2 here: they are significantly more difficult to use then plain assertions, but are not as good error-message quality wise as hand-crafted messages. And they don't really help with 3, where you need some kind of domain-specific diffing. EDIT: what works great as a midpoint between 1 & 3 is pytest / better_assertions style of thing, which generates reasonable diff from the plain assert.

So, TL;DR, the reason I've switched to assert_cli is to be able to do .stdout().contains("substring").

The reason why I want to write my own harness instead of assert_cli has to do with mostly technical issues:

• It doesn't handle current_dir nicely, so I have to write custom code anyway.
• When stdout does not match, it doesn't show stderr, which usually contains the clue as to why stdout is wrong :)

I also want to import a couple of niceties from Cargo's harness. The API I'd love to have will look like this:

// split by space to get a list of args. Not general, but covers 99% of cases
cmd("review-deps diff rand:0.6.0 rand:0.6.1")  
    // *rutime* arguments can be handled with the usual builder pattern
    .arg("-d").arg(&tmp_dir) 
    // a builder for expectation, not too fluent and easily extendable (b/c this lives in the same crate)
    .stdout_contains("< version = \"0.6.1\"") 
    // An optional debug helper. If streaming is enabled, all output from subprocess goes to the terminal as well. Super-valuable for eprintln! debugging 
    // .stream()
    // the call that actually runs the thing.
    // on error, it uncomnditionally prints process status, stdout & stderr, and only then a specific assert message. 
    .status(101)
    // that's it, but I can imagine this returns something you can use for fine-grained checking of the output, so
    // .output() # to get std::process::Output
    // .text_output() # to get something isomorphic to `Output`, but with Strings instead of Vec<u8>.

Hiya, after the switch for faster bin lookup (released in 0.11), writing code to assert on examples has become quite verbose, albeit workable (trade offs!).

use std::process::Command;

use assert_cmd::{cargo::CargoError, prelude::*};

#[test]
fn example_01_draw_text() -> Result<(), CargoError> {
    Command::cargo_example("01_draw_text")?
        .env("APP_DIR", env!("CARGO_MANIFEST_DIR"))
        .args(&["--timeout", "0"])
        .output()
        .unwrap()
        .assert()
        .success();

    Ok(())
}

use assert_cmd::{cmd::OutputError, prelude::*};
use escargot::CargoBuild;

#[test]
fn example_01_draw_text() -> Result<(), OutputError> {
    CargoBuild::new()
        .example("01_draw_text")
        .run()
        .expect("Failed to create `cargo` command")
        .command()
        .env("APP_DIR", env!("CARGO_MANIFEST_DIR"))
        .args(&["--timeout", "0"])
        .ok()?
        .assert()
        .success(); // chain is pretty long now

    Ok(())
}

Possible Solution

With the --message-format parameter, we get get cargo build to tell us the path to the example executable:

cargo build --examples -p application_ui --message-format json

// .. 323 lines of other artifacts
{"reason":"compiler-artifact","package_id":"application_ui 0.9.0 (path+file:///D:/work/gitlab/azriel91/autexousious/crate/application_ui)","target":{"kind":["example"],"crate_types":["bin"],"name":"01_draw_text","src_path":"D:\\work\\gitlab\\azriel91\\autexousious\\crate\\application_ui\\examples/01_draw_text/main.rs","edition":"2018"},"profile":{"opt_level":"0","debuginfo":2,"debug_assertions":true,"overflow_checks":true,"test":false},"features":[],"filenames":["D:\\work\\gitlab\\azriel91\\autexousious\\target\\debug\\examples\\01_draw_text.exe"],"executable":"D:\\work\\gitlab\\azriel91\\autexousious\\target\\debug\\examples\\01_draw_text.exe","fresh":true}

The relevant bit is:

"executable":"D:\\work\\gitlab\\azriel91\\autexousious\\target\\debug\\examples\\01_draw_text.exe"

I didn't figure out if cargo-metadata allows you to get similar information, but if we could get that information quickly (possibly within assert_cmd), and not require users to compile cargo, it's likely to be a nice solution.

edit:

ah, only just figured out what current_target() and current_release() meant (removed it from the comparison). Took a while to figure out that escargot is a library to construct a cargo command, and not a magic swap-in replacement for cargo.

predicates is going to split into predicates and predicates-core. Only predicates-core can show up in our public API.

This work was done for our IntoOutputPredicate predicates but the IntoCodePredicates were missed.

See https://github.com/assert-rs/assert_cmd/blob/master/src/assert.rs#L370

Since its in our public interface, we are blocked on it for going to 1.0

See also assert-rs/predicates-rs#30

cargo test takes a long time with because the first run of main_binary takes a long time while holding the cargo locking, blocking other tests

Example

running 6 tests
test blackbox_backup ... test blackbox_backup has been running for over 60 seconds
test blackbox_help ... test blackbox_help has been running for over 60 seconds
test blackbox_help ... ok
test blackbox_no_args ... ok
test clean_error_on_non_archive ... ok
test empty_archive ... ok
test incomplete_version ... ok
test blackbox_backup ... ok

From https://ci.appveyor.com/project/sourcefrog/conserve/builds/20036005

Cause

• If a bare cargo test is run, cargo uses target/debug for the OUT_DIR
• main_binary passes --target <TRIPLET> to cargo which changes the OUT_DIR to target/<TRIPLE>/debug
  • This means cargo tests build artifacts cannot be reused and those parts need to be rebuilt

Background:

In Issue #4 we had to choose between performance and correctness (if you do specify --target, main_binary should run that target's bin) and sided on correctness

Workarounds

1. Call into escargot without passing in current_target

2. Call cargo test --target <TRIPLE>

From @passcod in #74

For other docs-related things, see my comment on the assert_fs issue as this crate is structured very similarly and suffers from most of the same issues.

I'm assuming referring to

Foremost, while the rustdoc is good as a reference, the upfront documentation is severely lacking. The readme is practically useless both for figuring out what this is (beyond the title, which is only helpful if one knows what all the words mean), and figuring out what it can do (and hence, whether it can be used for one's usecase). Going to the docs.rs page in some hope of more is equally discouraging (and also non-obvious from github as it's only linked through the badge), as the crate top-level documentation is an extremely succinct "overview" that doesn't really explain anything.

See assert-rs/assert_fs#46

No idea if it doesn't actually work, or if I'm doing something wrong and API is simply allowing for it, but either way it should be fixed.

assert_cmd = "0.6"

my test case:

#[test]
fn broek_dep() {
    Command::new("cat")
        .assert()
        .set_stdin(vec![0])
        .stdout("\0")
        .success();
}

Which results in:

thread 'broek_dep' panicked at 'Unexpected stdout
command=`"cat"`
stdin=``````
code=0
stdout=``````
stderr=``````
',

Also on a side note, output of assert_cmd when there's an error is rather confusing and not helpful when you're trying to deal with input/output that consists of raw, often unprintable, bytes. Debug output without additional ``` would work a lot better IMO.

This provides something like the macro in assert_cli, so this could be helpful for people to transition

https://github.com/tcr/commandspec/blob/master/README.md

Right now, str gets turned into a Utf8<Difference> set of predicates. Should we add trimming and line ending normalization to that?

assert_cli did trimming. This would allow people to work around it by directly passing in a predicate rather than relying on IntoOutputPredicate.

I'm running into what I think is a bug with with_stdin, but I also could be misunderstanding the use-case for it!

I have boiled down my usage, to something where I can reproduce with just cat

let mut cmd = Command::new("cat");
cmd
    .with_stdin()
    .buffer("42");
let assert = cmd.assert();
assert
  .success()
  .stdout("42");

I expected this to function kinda like echo 42 | cat which outputs 42

However this assert is not passing for me. Am I misunderstanding something, or is this a bug? I am using version 0.11.0 right now

Thanks for the time, and for writing and maintaining this library!

(Not sure if this should be an escargot question...)

I'm writing integration tests for different feature flags.

This is how I'm setting up the test env:

impl TestEnv {
    fn new() -> Self {
        TestEnv {
            cache_dir: TempDir::new(".tldr.test.cache").unwrap(),
            config_dir: TempDir::new(".tldr.test.config").unwrap(),
            input_dir: TempDir::new(".tldr.test.input").unwrap(),
            default_features: true,
            features: vec![],
        }
    }

    /// Disable default features.
    fn no_default_features(mut self) -> Self {
        self.default_features = false;
        self
    }

    /// Add the specified feature.
    fn with_feature<S: Into<String>>(mut self, feature: S) -> Self {
        self.features.push(feature.into());
        self
    }

    /// Return a new `Command` with env vars set.
    fn command(&self) -> Command {
        let mut build = escargot::CargoBuild::new()
            .bin("tldr")
            .current_release()
            .current_target();
        if !self.default_features {
            build = build.arg("--no-default-features");
        }
        if !self.features.is_empty() {
            build = build.arg(&format!("--feature {}", self.features.join(",")));
        }
        let run = build.run().unwrap();
        let mut cmd = run.command();
        println!("Command: {:?}", cmd);
        cmd.env("TEALDEER_CACHE_DIR", self.cache_dir.path().to_str().unwrap());
        cmd.env("TEALDEER_CONFIG_DIR", self.config_dir.path().to_str().unwrap());
        cmd
    }
}

I can then invoke the binary with various feature configs like this:

TestEnv::new().with_feature("logging").command()...
TestEnv::new().with_feature("networking").command()...
TestEnv::new().no_default_features().command()...

The problem is that when tests run in parallel, some of them randomly fail. They pass if I run them sequentially.

The way I understand it, escargot runs cargo for the specified target. It's cargo's job to determine whether to rebuild or not.

If the arguments change, then of course cargo will do a rebuild. But since tests run in parallel, it's possible that the "wrong" binary is used while the rebuild is still in process...

Is there a way to use a separate build directory per feature flag configuration? Or maybe you could add some kind of opt-in mutex feature to assert_cmd that enforces assert_cmd based tests to run sequentially?

Right now you have to hunt through all of the traits and it makes it hard to see the big picture.

assert-rs/predicates-rs#7

See https://crates.io/crates/version-sync

I'm trying to use this for a project instead of the published version of assert_cli. I'm super impressed by what you've accomplished here in the short time!

It seems that currently, there is no easy way to check that the output of a command contains some text. Or did I miss something?

Predicates only has a str::contains but stdout is a Vec<u8>. It seems that it could work with predicate::function, but it is quite cumbersome. I'm basically looking for good way of writing .stdout(&predicate::function(|x: &Vec<u8>| String::from_utf8_lossy(x).contains("Done!"))) -- but ideally without the need to use from_utf8_lossy. This last requirement is also why I opened this issue here, and not on the predicates, I expect we may need to talk about how to best deal with the Vec<u8> representation.

• code: take IntoCodePredicate
  • From<i32>
  • From<Predicate<i32>>
• stdin / stdout: take IntoOutputPredicate
  • From<str>
  • From<Predicate<str>>
  • From<[u8]>

Sometimes we want to provide testing for different features bin.

Steps to reproduce:

• write an integration test that uses the main binary like this: ...Command::cargo_bin("cli-name").unwrap()
• run the tests
• rename the crate in Cargo.yaml (thus renaming the binary)
• break some code
• run the tests again

The tests pass! One easy solution is to use env!(CARGO_PKG_NAME), but I'm more concerned about it having used a stale cache without even noticing it was doing the "wrong thing".

I can see how this happens and that this crate is not doing anything wrong, but I just wanted to point this out.

Things to cover

• Command is run at assert or earlier and not at the end of the expression
• Output predicates don't automatically trim (unless we change it, see #19)
• success is not automatically asserted.

For example, I need to match:

lib_crate::sum_array (src/lib.rs:6):
 push    rbp
 mov     rbp, rsp
 test    rsi, rsi
  je      LBB13_1
  lea     r9, [4*rsi, -, 4]
 shr     r9, 2
 inc     r9
 cmp     r9, 8
 jae     LBB13_4
  xor     eax, eax
 mov     rcx, rdi
 jmp     LBB13_13
LBB13_1:
 xor     eax, eax
 pop     rbp
 ret

on macos, but on windows all the LBB13_... are LBB14_.. and they change on linux as well. Also on windows and linux, the push+mov at the beginning, and pop at the end parts of my pattern are not present.

In rustc, I would write something like this

lib_crate::sum_array (src/lib.rs:6):
{{.*}}
 test    rsi, rsi
 je      LBB{{.*}}_1
 lea     r9, [4*rsi, -, 4]
 shr     r9, 2
 inc     r9
 cmp     r9, 8
 jae     LBB{{.*}}_4
 xor     eax, eax
 mov     rcx, rdi
 jmp     LBB{{.*}}_13
LBB{{.*}}_1:
 xor     eax, eax
{{.*}}
 ret

where the {{regexp}} is used to match any pattern.

What's the easiest way of doing something like this to assert outputs that using assert_cmd ?

It isn't always clear what you can do.

@mssun

Another issue about testing CLIs is to correctly calculate code coverage. If using std::process::Command to fork and execute a CLI, all existing code coverage tools (tarpaulin, gcov, kcov, etc) will failed to calculate the statistics. There is not a better way to properly calculate the line coverage including unit tests and integration tests.

From https://github.com/rust-lang-nursery/cli-wg/issues/9#issuecomment-385487756

It will be very useful if I can use a fixture file as input

The implementations are only for 'static. Can we do better?

See https://docs.rs/assert_cmd/0.9.1/assert_cmd/assert/trait.IntoOutputPredicate.html

As food for thoughts, here's how I'm using assert_cmd.

The guiding idea is that I build a struct describing programs to run/delay/kill, a set of input files, and a set of post-run checks that operate on created files (including the program's stdout/stderr). I can run() on that struct once it is set up, and it takes care of setting up the TempDir, scheduling program start, and running the checks.

/// Main struct described above. It's got builder-pattern methods,
/// and a `run` method that will do all the work and `assert` upon failure.
pub struct Exec {
    cmds: Vec<Cmd>,
    timeout: Duration,
    files: Vec<FileIn>,
    checks: Vec<FilePredicate>,
}

pub struct Cmd {
    /// Program to run.
    bin: Command,
    /// Command name for `after()`, std/err filname prefix, and logs.
    name: String,
    /// Expected exit status. Some(0)=success, Some(n)=failure, None=timeout.
    exit: Option<i32>,
    /// Fail if the cmd exits too early.
    mintime: Duration,
    /// Fail if the cmd run for too long.
    maxtime: Duration,
    /// Current state.
    state: CmdState,
    /// List of signals to send to the process after startup.
    signals: Vec<(CmdCond, c_int)>,
}

enum CmdState {
    Wait(CmdCond),
    Started(Child, Instant),
    Done,
}

pub enum CmdCond {
    /// Immediately true
    None,
    /// Duration elapsed
    Delay(Duration),
    /// Other Cmd exited
    Cmd(String),
    /// File-based predicate
    Predicate(FilePredicate),
}

pub struct FilePredicate {
    /// Desciption for assert-logging purpose.
    desc: String,
    /// Which file to operate on.
    file: String,
    /// Closure that tests the content of the file.
    pred: Box<dyn Fn(&str) -> bool>,
}

pub enum FileIn {
    FromFs(&'static str, &'static str),
    Bin(&'static str, Vec<u8>),
}

With that in place, I have a pretty powerful way to write a unittest, using my crate's main binary and a few ancillary processes:

    // As basic as it gets.
    exec().cmd(Cmd::any("echo", "echo", "-n a")).check("echo a", "echo.out", |s| s == "a").run();
    // File from hardcoded data.
    exec().inbytes("a", "a")
          .cmd(Cmd::any("cat", "cat", "a"))
          .check("cat a", "cat.out", |s| s == "a")
          .run();
    // File from file in test/ directory.
    exec().infile("j", "input.basic.json")
          .cmd(Cmd::any("cat", "cat", "j"))
          .check("cat j", "cat.out", |s| s.starts_with("{"))
          .run();
    // run sequentially
    let start = Instant::now();
    exec().cmd(Cmd::any("s1", "sleep", "0.3"))
          .cmd(Cmd::any("s2", "sleep", "0.3").after("s1"))
          .cmd(Cmd::any("s3", "sleep", "0.3").after("s2"))
          .cmd(Cmd::any("cat", "cat", "s1.out s2.out s3.out").after("s3"))
          .run();
    assert!(Instant::now() - start > Duration::from_millis(900));
    // delayed start
    exec().cmd(Cmd::any("0", "cat", "1.out 2.out").after(20))
          .cmd(Cmd::any("1", "echo", "a"))
          .cmd(Cmd::any("2", "echo", "b"))
          .check("cat", "0.out", |s| s == "a\nb\n")
          .run();
    // timeout
    exec().cmd(Cmd::any("s", "sleep", "1").maxtime(100).exit(None)).run();
    exec().cmd(Cmd::any("s", "sleep", "0.05").mintime(50).maxtime(70)).run();
    // Multiple signals, ordered by Instant.
    exec().cmd(Cmd::any("s", "sleep", "1").signal(SIGINT, 70) // Added first but triggers last
                                          .signal(SIGCONT, 10) // Triggers first but doesn't terminate
                                          .signal(SIGTERM, 30) // Actual terminator
                                          .exit(SIGTERM))
          .run();
    // Signal after another cmd exit.
    exec().cmd(Cmd::any("s1", "sleep", "1").signal(SIGINT, "s2").maxtime(50).exit(SIGINT))
          .cmd(Cmd::any("s2", "sleep", "0.01"))
          .run();
    // Signal after file content matches
    exec().cmd(Cmd::any("s1", "sleep", "1").signal(SIGINT, ("s2.out", "4"))
                                           .maxtime(100)
                                           .exit(SIGINT))
          .cmd(Cmd::bash("s2", "for i in $(seq 10);do echo $i;sleep 0.01;done"))
          .timeout(1000)
          .run();
    // Main binary
    exec().cmd(Cmd::main("m", "-V")).check("progname", "0.out", |s| s.starts_with("mybin"));
    // Set/unset env
    exec().cmd(Cmd::any("env", "env", "").env("foo", "bar")
                                         .env("PATH", &format!("/ut:{}", env::var("PATH").unwrap()))
                                         .env_remove("HOME"))
          .check("added_foo", "env.out", |s| s.lines().any(|l| l == "foo=bar"))
          .check("changed_path", "env.out", |s| s.lines().any(|l| l.starts_with("PATH=/ut:")))
          .check("removed_home", "env.out", |s| !s.lines().any(|l| l.starts_with("HOME=")))
          .run();

If there's enough interest, I could try to get this cleared for upstreaming. Currently there are a few extensions and API decisions that are specific to my project, and the code is a bit ad-hoc in places.

Originally posted by @vincentdephily in #74 (comment)

Use cases

• Specifying features (see #34)
• Depending on which we keep as default, either (see #4)
  • Specify the target to get cross-compilation coverage and improve cross-compilation test performance
  • Don't specify the target to improve naive-compilation test performance

Considerations

• The primary use case of assert_cmd is testing Rust binaries, so we should help those users
• An entire cargo API is orthogonal to assert_cmd and would be best to be stablized outside of assert_cmd

Proposal

• Mirror the concept of cargo run in escargot but via CargoBuild
  • CargoBuild would have a .run() method that would return a CargoRun
  • CargoRun would have .path() and .command() methods.
• Still provide CommandCargoExt but point people to escargot for any custom behavior
• Deprecate the cargo path functions and instead point people to escargot for those use cases.
  • Create an issue for them to be deleted as part of the 1.0 milestone

At this point, CommandCargoExt is primarily serving a low-effort path for a "getting started" user while serving as an example of how to do more complex things (via code) and pointing to more complex APIs (via docs)

Open Issues

• What should be the default for #4

Collecting feedback or assert_fs. Please post even if you think it might not be needed for 1.0 so we can try to get the big picture of uses, challenges, etc. We're also looking at including experience reports in the 1.0 announcement blog post.

Preferably, please update to the latest version first but don't let upgrading stop you from providing feedback

To help spur feedback, consider these questions

• Why do you use this crate?
• Why don't you use it more?
• How does this crate fit into your testing strategy?
• Is there anything that feels out of place / not rust-like?
• Are there common patterns you implement on top of this crate (e.g. fixture setup or assertions we don't support)?

Also, if your programs deals with the file system, please provide feedback on assert_fs. Even if you don't use it, I want to know why.

Summary

Areas to improve

• #76 Improve overview / introduction in README / docs.rs
• #75 Highlight Assert::get_output in docs
• #73 stdin handling is an unfriendly API

Successes

fitzgen

assert_cmd is just such a pleasure to use every single time, I fall in love all over again
bravo bravo WG-cli

@passcod

Running commands and dealing with output can be complex in many many ways, so assert_cmd smoothing that is excellent, very much welcome, and improves ergonomics significantly.

@volks73

I have used [assert_cmd] in other projects and I am extremely pleased with it

@coreyja

[assert_cmd] pretty much IS my testing strategy so far, though my app under test is pretty small.

This library has made it really easy to add some test coverage to my project, even when I am just learning how to write Rust!

From @rask assert-rs/assert_cli#101

workaround: manually call binaries under test, see assert-rs/assert_cli#101 (comment)

• assert_cli version: 0.5.4
• Rust version: Stable, 1.25.0
• OS and version: Linux Ubuntu 17.10

I have a CLI application in the works and wanted to see if assert_cli was of any use for me. The crate works fine when running cargo test. Neat stuff. :)

After running cargo test I use kcov to generate code coverage reports for my code. When assert_cli is in use kcov makes the tests fail with a signal: 11, SIGSEGV: invalid memory reference.

I'm not sure if this is related to my newbie Rust code or if kcov and assert_cli just don't like each other for some reason.

My code for the test:

// tests/lib_test.rs
extern crate assert_cli;

use std::env;

fn get_cwd() -> String {
    env::current_dir().unwrap().to_str().unwrap().to_string()
}

#[test]
fn test_app_config_works() {
    let cfg_file: &str = &format!("{}/tests/data/libtestwppr.toml", get_cwd());

    assert_cli::Assert::main_binary()
        .with_args(&["help"])
        .succeeds()
        .stdout()
        .contains("list")
        .unwrap();

    assert_cli::Assert::main_binary()
        .with_args(&["--configuration", cfg_file, "list"])
        .succeeds()
        .stdout()
        .contains("plugin.php")
        .stdout()
        .contains("0.1.2")
        .unwrap();
}

And the error I get with RUST_BACKTRACE=full:

failures:                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                              
---- test_app_config_works stdout ----                                                                                                                                                                                                        
        thread 'test_app_config_works' panicked at 'CLI assertion failed: (command `cargo run -- --configuration /home/ojrask/Projects/ojrask/wp-plugin-repofier/tests/data/libtestwppr.toml help` expected to succeed)                       
status=failed                                                                                                                                                                                                                                 
stdout=``````                                                                                                                                                                                                                                 
stderr=```error: process didn't exit successfully: `rustc -vV` (signal: 11, SIGSEGV: invalid memory reference)                                                                                                                                
```', /home/ojrask/.cargo/registry/src/github.com-1ecc6299db9ec823/assert_cli-0.5.4/src/assert.rs:368:13                                                                                                                                      
stack backtrace:                                                                                                                                                                                                                              
   0:     0x5555556cc6f3 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h09c1ee131a74b1c4                                                                                                                                       
                               at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49                                                                                                                                                               
   1:     0x5555556c3924 - std::sys_common::backtrace::_print::h47a337b62b6d5e9e                                                                                                                                                              
                               at libstd/sys_common/backtrace.rs:71                                                                                                                                                                           
   2:     0x5555556c92fd - std::panicking::default_hook::{{closure}}::h945a649c9017832e                                                                                                                                                       
                               at libstd/sys_common/backtrace.rs:59                                                                                                                                                                           
                               at libstd/panicking.rs:380                                                                                                                                                                                     
   3:     0x5555556c8f9b - std::panicking::default_hook::hcc534c2d30fbcda3                                                                                                                                                                    
                               at libstd/panicking.rs:390                                                                                                                                                                                     
   4:     0x5555556c9770 - std::panicking::rust_panic_with_hook::h09a7a3a353dc2f38                                                                                                                                                            
                               at libstd/panicking.rs:576                                                                                                                                                                                     
   5:     0x5555556c962e - std::panicking::begin_panic::h8327f16bde15df70                                                                                                                                                                     
                               at libstd/panicking.rs:537                                                                                                                                                                                     
   6:     0x5555556c9529 - std::panicking::begin_panic_fmt::h42ff1d37404632d6                                                                                                                                                                 
                               at libstd/panicking.rs:521                                                                                                                                                                                     
   7:     0x5555555b667d - assert_cli::assert::Assert::unwrap::h0b7b03819134f9eb                                                                                                                                                              
                               at /home/ojrask/.cargo/registry/src/github.com-1ecc6299db9ec823/assert_cli-0.5.4/src/assert.rs:368                                                                                                             
   8:     0x55555557256c - lib_test::test_app_config_works::h1f9913950e84dc3c                                                                                                                                                                 
                               at tests/lib_test.rs:13                                                                                                                                                                                        
   9:     0x55555557f7f1 - <F as alloc::boxed::FnBox<A>>::call_box::h8106479744bcbe3a                                                                                                                                                         
                               at libtest/lib.rs:1462                                                                                                                                                                                         
                               at /checkout/src/libcore/ops/function.rs:223                                                                                                                                                                   
                               at /checkout/src/liballoc/boxed.rs:788                                                                                                                                                                         
  10:     0x5555556e1efe - __rust_maybe_catch_panic                                                                                                                                                                                           
                               at libpanic_unwind/lib.rs:102                                                                                                                                                                                  
  11:     0x555555577063 - std::sys_common::backtrace::__rust_begin_short_backtrace::hbe6e8f3a8f0dc6f8                                                                                                                                        
                               at /checkout/src/libstd/panicking.rs:458                                                                                                                                                                       
                               at /checkout/src/libstd/panic.rs:358                                                                                                                                                                           
                               at libtest/lib.rs:1414                                                                                                                                                                                         
                               at /checkout/src/libstd/sys_common/backtrace.rs:136                                                                                                                                                            
  12:     0x555555578162 - std::panicking::try::do_call::he62ddbc80b0afa86                                                                                                                                                                    
                               at /checkout/src/libstd/thread/mod.rs:406                                                                                                                                                                      
                               at /checkout/src/libstd/panic.rs:293                                                                                                                                                                           
                               at /checkout/src/libstd/panicking.rs:479                                                
  13:     0x5555556e1efe - __rust_maybe_catch_panic        
                               at libpanic_unwind/lib.rs:102                                                           
  14:     0x55555557f582 - <F as alloc::boxed::FnBox<A>>::call_box::h4290b0f48d980187
                               at /checkout/src/libstd/panicking.rs:458
                               at /checkout/src/libstd/panic.rs:358
                               at /checkout/src/libstd/thread/mod.rs:405
                               at /checkout/src/liballoc/boxed.rs:788
  15:     0x5555556d962b - std::sys::unix::thread::Thread::new::thread_start::h711c51a13a158afa
                               at /checkout/src/liballoc/boxed.rs:798
                               at libstd/sys_common/thread.rs:24
                               at libstd/sys/unix/thread.rs:90
  16:     0x7ffff71ae7fb - start_thread
  17:     0x7ffff6cc4b5e - clone
  18:                0x0 - <unknown>


failures:
    test_app_config_works

As I stated: the tests pass just fine when I run a plain cargo test instead of using kcov. kcov operates by taking the built test binaries and running them through again, while looking at what is being run. Would this pose a problem for kcov or assert_cli which is used to run "external" bins that are not directly built from the test case?

When I remove the lib_test.rs file from kcov runs the tests pass for kcov again, but nothing relating to assert_cli is reported as covered.

The application I'm working on is over at https://github.com/rask/wppr which contains the logic right now, the above tests are the only thing I've added which are not in the repo. You can see the coverage.sh for a script which I use to run coverage tests on my code. (The in-repo coverage.sh is missing the kcov definition for running the above lib_test.rs file, but it is the same as the wordpress_test- pattern inclusion in there.)

Please let me know if this issue should be moved somewhere else. :)

From @passcod on #74

It's not obvious that to get the output of a command without doing asserts on it (e.g. if I want to use another assertion tool) I have to still go through the Assert trait. I ended up writing a helper to read stdout and stderr to string via the Command handle but later saw I could have just used get_output().stdout.to_string()...

I'm thinking we should have a note in the assert documentation mentioning the ability to access this for completely custom logic outside of the scope of assert_cmd

Based on #71

The automatic reaction is to write something like

let mut cmd = Command::new("cat");
cmd
    .with_stdin()
    .buffer("42");
let assert = cmd.assert();
assert
  .success()
  .stdout("42");

When you really need to write

 let mut cmd = Command::new("cat");
 cmd
     .arg("-et");
 cmd
     .with_stdin()
     .buffer("42")
     .assert()
     .stdout("42");

The problem is an issue with &mut builders (we can't chain off the original cmd), assuming with_stdin will act like &mut builder function when in reality you should only be acting on it at that point, etc.

Note: The lib.rs documentation still has the broken approach

This also brings it to parity with assert_cli