camshaft / bolero Goto Github PK

View Code? Open in Web Editor NEW

167.0 4.0 14.0 2.71 MB

property testing and verification front-end for Rust

Home Page: https://camshaft.github.io/bolero

License: MIT License

Makefile 1.73% Rust 16.92% CMake 0.56% C++ 25.91% C 54.55% Shell 0.19% Python 0.14%

fuzz fuzz-testing property-testing libfuzzer honggfuzz afl

bolero's People

Contributors

Stargazers

Watchers

Forkers

ckaran g2p zhassan-aws stjordanis ekleog matklad link2xt ekleog-near adpaco-aws binadamu-isiyoonekana stranger6667 glebpom valenrukra celinval

bolero's Issues

Generator for floating point within a range?

Does bolero support generating floating point values with a restricted range?

gen::<f64>().with()
            .bounds(0.0,,1.0)

Doesn't seem to work.

impl TypeGenerator for chrono::DateTime, chrono_tz::Tz, etc?

I'm wondering, where do you think TypeGenerator impls for widely-used crates should live? I just tried to add #[derive(TypeGenerator)] to a crate of mine that uses chrono, and had to fall back to using arbitrary after noticing impls of TypeGenerator for chrono types don't seem to exist yet.

Relatedly, do you think a fallback like the following would make sense?

#[derive(TypeGenerator)]
struct Foo {
    bar: usize,
    #[type_generator(from_arbitrary)]
    baz: BazIsArbitraryButNotTypeGenerator,
}

Or maybe it's too much effort to be worth it, and it'd be better to just spend that effort upstreaming bolero-generator everywhere

Switching to a magic number-based collection generator?

Hey!

I've just recently had a look at the TypeGenerator for Vec, and noticed that the collection generators seem to use a tag-length-value format.

Such a format is very quickcheck-friendly (because all inputs are valid), but it is not at all fuzzer-friendly, because a change of one in the length of one struct is going to completely mess with the whole structure of the rest of the input.

Google recommends a magic-number-based parser, as per https://github.com/google/fuzzing/blob/master/docs/split-inputs.md#magic-separator : instead of having a length, have a separator used to go to the next input.

With this scheme, quickcheck is much less happy (because it needs to somehow figure out this magic number), but fuzzers are much happier (because they can easily change the length of the contents of one cell without messing with the rest).

So I'm thinking, do you think it'd make sense to change the generators of cargo-bolero so that they would use a magic-number scheme? The only issue I can see would be with the quickcheck-based backend, which I think could instead always use mutate so it doesn't have the issue that most inputs would have a single element in the collection (unless the magic number magically appeared)

Construct fuzzer input from known result (AntiGenerator?)

This may be a far-fetched idea, but I'm just dropping it here in case it'd make sense: what would you think about the idea of making an "anti-generator" function in the generator traits, that'd look like:

trait TypeGenerator {
    // ... all there already is
    fn as_test_case(&self) -> Vec<u8>;
}

Then, one could use this generate_test_case funciton so as to build a crash fuzzer test from a known-bad value of that type.

TBH I'm not sure it's actually worth the hassle (as theoretically just running the fuzzer should find the input automagically), but I wanted to drop this idea so we could discuss it :)

Add support for crux engine

https://github.com/GaloisInc/crucible/tree/master/crux-mir

$ cargo bolero fuzz my_test --fuzzer crux

fn my_test() {
    bolero::fuzz!()
        .with_type::<u32>()
        .for_each(|value| {
            assert_ne!(value, 12345678);
        })
}

Roughly equivalent to

#[crux_test]
fn my_test() {
    let value = u32::symbolic("value");
    assert_ne!(value, 12345678);
}

Trophy case?

Hey!

I just incidentally found a panic inside chrono, while fuzzing my web server: chronotope/chrono#941

So I'm wondering if you'd want to introduce a trophy case to cargo-bolero, similar to the one cargo-fuzz has? :)

Termination condition of `check!()`?

I've rewritten most of my unit tests to use check!() at this point, and have run into an interesting problem; a couple of my tests aren't terminating. I put in a print!("{:?}: {:?}: date = {:?}\r", file!(), line!(), std::time::SystemTime::now()); call at the start of the closure for check!(), so I know that it is making progress (not a hung process), but it can take several hours before these particular unit tests terminate. So, how does check!() decide to terminate? Can we limit its runtime, e.g. by passing in a std::time::Duration, and have the test time out if it exceeds that value?

`cargo bolero test --engine kani <test>` fails to match harness

There is some issue in selecting the proper integration test for Kani:

$ cargo bolero --version
cargo-bolero 0.8.0

$ cargo kani --version
cargo-kani 0.23.0

$ cargo new project

$ cargo bolero new verify

$ cargo bolero test --engine kani verify
error: no harnesses matched the harness filter: `verify`

$ cat tests/verify/main.rs
#[cfg_attr(kani, kani::proof)]
fn main() {
    bolero::check!().for_each(|v| {
        let _ = todo!();
    });
}

The following works, but executes all test targets:

$ cargo kani --tests

The following works, but isn't very useful:

$ cargo kani --tests --harness main

The following works, but doesn't use Kani:

$ cargo bolero test verify

Is this a problem with cargo kani rather than cargo bolero?

P.S: actual setup is within a cargo workspace

Feature request: make fuzz!() usable in unit tests

I'm slowly converting my code to using bolero from using proptest, mainly because I think bolero has better fuzz testing. However, I'd like to be able to use fuzz!() within unit tests (I know about check!(), but since it isn't fuzzing, I'm not as interested in it). I know that this doesn't work right now; when I try to use fuzz!() within even simple unit tests I get a segfault. But since I know very little about how bolero works under the hood, I don't know if this is something that is easy to fix, or if it will be a major effort. So, would it be easy to get fuzz!() to work correctly within unit tests? If not, can you please tell me why, just to satisfy my curiosity?

fuzz!() doesn't work in release mode.

The complete information for this is in MR #15, but the gist is that cargo test and cargo test --release have very different results. In the former, things work, but in the latter, I get the following error:

thread 'tests::new' panicked at 'test name not found', /home/cfkaran2/.cargo/registry/src/github.com-1ecc6299db9ec823/bolero-engine-0.5.1/src/target_location.rs:92:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    tests::new

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--lib'

Use try_from_fn for array generator implementation once stable

rust-lang/rust#89379

This will clean up the implementation quite a bit

Passing options to the fuzzers

I need to pass a new timeout value to honggfuzz as 1 second is too little. Running cargo test, I get stuck on types that never complete. Running cargo bolero fuzz -f honggfuzz --release nodes::behaviors::trading_post::datum::tests::hashes yields the following, which is rejecting a lot of possible tests:

cargo bolero fuzz -f honggfuzz --release nodes::behaviors::trading_post::datum::tests::hashes
    Finished release [optimized] target(s) in 0.07s
Persistent signature found in '/home/cfkaran2/Documents/repositories/bit_network/target/fuzz/build_a6f025e43af3afe0/x86_64-unknown-linux-gnu/release/deps/bit_network_library-83616d3b5ba9aab8'. Enabling persistent fuzzing mode
Start time:'2020-05-26.11.46.11' bin:'/home/cfkaran2/Documents/repositories/bit_network/target/fuzz/build_a6f025e43af3afe0/x86_64-unknown-linux-gnu/release/deps/bit_network_library-83616d3b5ba9aab8', input:'/home/cfkaran2/Documents/repositories/bit_network/bit_network_library/src/nodes/behaviors/trading_post/__fuzz__/nodes__behaviors__trading_post__datum__tests__hashes/corpus', output:'/home/cfkaran2/Documents/repositories/bit_network/bit_network_library/src/nodes/behaviors/trading_post/__fuzz__/nodes__behaviors__trading_post__datum__tests__hashes/corpus', persistent:true, stdin:false, mutation_rate:5, timeout:1, max_runs:0, threads:4, minimize:false, git_commit:71c712fd2cc2c15e545ca447fc9219a246be82fb







------------------------[  0 days 00 hrs 00 mins 54 secs ]----------------------
  Iterations : 20,444 [20.44k]
  Mode [3/3] : Feedback Driven Mode
      Target : /home/cfkaran2/Documents/reposit.....re --quiet --test-threads 1
     Threads : 4, CPUs: 8, CPU%: 472% [59%/CPU]
       Speed : 0/sec [avg: 378]
     Crashes : 0 [unique: 0, blacklist: 0, verified: 0]
    Timeouts : 196 [1 sec]
 Corpus Size : 483, max: 8,192 bytes, init: 799 files
  Cov Update : 0 days 00 hrs 00 mins 01 secs ago
    Coverage : edge: 1,548/514,933 [0%] pc: 1,436 cmp: 25,733
---------------------------------- [ LOGS ] ------------------/ honggfuzz 2.2 /-
Persistent mode: Launched new persistent pid=906
Persistent mode: Launched new persistent pid=907
Persistent mode: Launched new persistent pid=908
Persistent mode: Launched new persistent pid=909
[2020-05-26T11:47:03-0400][W][368] subproc_checkTimeLimit():530 pid=907 took too much time (limit 1 s). Killing it with SIGKILL
[2020-05-26T11:47:03-0400][W][369] subproc_checkTimeLimit():530 pid=906 took too much time (limit 1 s). Killing it with SIGKILL
[2020-05-26T11:47:03-0400][W][370] subproc_checkTimeLimit():530 pid=909 took too much time (limit 1 s). Killing it with SIGKILL
[2020-05-26T11:47:03-0400][W][371] subproc_checkTimeLimit():530 pid=908 took too much time (limit 1 s). Killing it with SIGKILL
[2020-05-26T11:47:03-0400][W][368] arch_checkWait():235 Persistent mode: pid=907 exited with status: SIGNALED, signal: 9 (Killed)
[2020-05-26T11:47:03-0400][W][370] arch_checkWait():235 Persistent mode: pid=909 exited with status: SIGNALED, signal: 9 (Killed)
[2020-05-26T11:47:03-0400][W][371] arch_checkWait():235 Persistent mode: pid=908 exited with status: SIGNALED, signal: 9 (Killed)
[2020-05-26T11:47:03-0400][W][369] arch_checkWait():235 Persistent mode: pid=906 exited with status: SIGNALED, signal: 9 (Killed)
Persistent mode: Launched new persistent pid=918
Persistent mode: Launched new persistent pid=919
Persistent mode: Launched new persistent pid=920
Persistent mode: Launched new persistent pid=921
Sz:20 Tm:38us (i/b/h/e/p/c) New:0/0/0/0/0/3, Cur:0/0/0/0/0/10
[2020-05-26T11:47:04-0400][W][369] subproc_checkTimeLimit():530 pid=921 took too much time (limit 1 s). Killing it with SIGKILL
[2020-05-26T11:47:04-0400][W][370] subproc_checkTimeLimit():530 pid=919 took too much time (limit 1 s). Killing it with SIGKILL
[2020-05-26T11:47:04-0400][W][368] subproc_checkTimeLimit():530 pid=918 took too much time (limit 1 s). Killing it with SIGKILL
[2020-05-26T11:47:04-0400][W][371] subproc_checkTimeLimit():530 pid=920 took too much time (limit 1 s). Killing it with SIGKILL
[2020-05-26T11:47:04-0400][W][369] arch_checkWait():235 Persistent mode: pid=921 exited with status: SIGNALED, signal: 9 (Killed)
[2020-05-26T11:47:04-0400][W][368] arch_checkWait():235 Persistent mode: pid=918 exited with status: SIGNALED, signal: 9 (Killed)
[2020-05-26T11:47:04-0400][W][370] arch_checkWait():235 Persistent mode: pid=919 exited with status: SIGNALED, signal: 9 (Killed)
[2020-05-26T11:47:04-0400][W][371] arch_checkWait():235 Persistent mode: pid=920 exited with status: SIGNALED, signal: 9 (Killed)
Persistent mode: Launched new persistent pid=923
Persistent mode: Launched new persistent pid=924
Persistent mode: Launched new persistent pid=925
Persistent mode: Launched new persistent pid=926
Sz:32 Tm:80us (i/b/h/e/p/c) New:0/0/0/0/0/3, Cur:0/0/0/0/0/13
Sz:512 Tm:472us (i/b/h/e/p/c) New:0/0/0/0/0/1, Cur:0/0/0/0/0/16
[2020-05-26T11:47:05-0400][W][368] subproc_checkTimeLimit():530 pid=924 took too much time (limit 1 s). Killing it with SIGKILL
[2020-05-26T11:47:05-0400][W][369] subproc_checkTimeLimit():530 pid=923 took too much time (limit 1 s). Killing it with SIGKILL
[2020-05-26T11:47:05-0400][W][370] subproc_checkTimeLimit():530 pid=925 took too much time (limit 1 s). Killing it with SIGKILL
[2020-05-26T11:47:05-0400][W][371] subproc_checkTimeLimit():530 pid=926 took too much time (limit 1 s). Killing it with SIGKILL
[2020-05-26T11:47:05-0400][W][369] arch_checkWait():235 Persistent mode: pid=923 exited with status: SIGNALED, signal: 9 (Killed)
[2020-05-26T11:47:05-0400][W][370] arch_checkWait():235 Persistent mode: pid=925 exited with status: SIGNALED, signal: 9 (Killed)
[2020-05-26T11:47:05-0400][W][371] arch_checkWait():235 Persistent mode: pid=926 exited with status: SIGNALED, signal: 9 (Killed)
[2020-05-26T11:47:05-0400][W][368] arch_checkWait():235 Persistent mode: pid=924 exited with status: SIGNALED, signal: 9 (Killed)
Persistent mode: Launched new persistent pid=927
Persistent mode: Launched new persistent pid=928
Persistent mode: Launched new persistent pid=929
Persistent mode: Launched new persistent pid=930
Signal 2 (Interrupt) received, terminating
Terminating thread no. #0, left: 3
Terminating thread no. #2, left: 2
Terminating thread no. #3, left: 1
Terminating thread no. #1, left: 0
Summary iterations:20444 time:54 speed:378 crashes_count:0 timeout_count:196 new_units_added:63 slowest_unit_ms:1169 guard_nb:514933 branch_coverage_percent:0 peak_rss_mb:59

Advice on debugging a failure.

I need some advice on how to track down a failure that I strongly suspect is coming from bolero. I have 3 tests that consistently fail, but they don't produce any failure output; they just get marked as failures. My suspicion is that its because they are running for more than 60 seconds, and that deep in the bowels of bolero they aren't producing enough output, so that's why the test is failing. However, I don't have a good way of verifying this, and there is too much code for me to produce a reduced example. Do you have any suggestions?

Here is an example of the cleaned up output:

cfkaran2@Hammer:~/Documents/repositories/bit_network$ RUST_BACKTRACE=full cargo test -- configuration_parsing::scenario_parser::tests::_config_tests::try_from_scenario

<Removed all the compilation results>

running 1 test
test configuration_parsing::scenario_parser::tests::_config_tests::try_from_scenario ... FAILED

failures:

failures:
    configuration_parsing::scenario_parser::tests::_config_tests::try_from_scenario

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 227 filtered out

error: test failed, to rerun pass '--lib'
cfkaran2@Hammer:~/Documents/repositories/bit_network$

This happens in both debug and release mode. I have all debug symbols on. There actually are two lines with failures in it, that isn't a copy/paste error.

Configuring `with_type`

I have a test that is similar to the following:

#[test]
fn sort() {
    bolero::check!()
        .with_type::<Vec<ProtocolMessage>>()
        .for_each(|mut golden| {
            // tests
    }
}

Is there a way of setting how long the vector is? Ideally I'd be able to feed in a range and bolero would handle the rest.

Add support for Quickcheck generators

https://docs.rs/quickcheck/0.9.2/quickcheck/trait.Arbitrary.html

Possible APIs:

fn test() {
    bolero::fuzz!()
        .with_quickcheck::<u64>()
        .for_each(|_| {
            // assertions here
        })
}

fuzz!() never terminates

This may be related to #10, but I don't have a good way of verifying.

I've upgraded to bolero 0.5.0, and replaced all of the calls to check!() with fuzz!(). Now some of my tests no longer terminate. I've let them run overnight to see if they would eventually terminate, but they didn't. Is there any way for me to capture some kind of output or log to attach to this bug report so you can look at it?

Deriving TypeGenerator requires that it is in scope

I usually do not use the derive-macros that I use, so that I always know which crate they're coming from.

However, if I #[derive(bolero_generator::TypeGenerator)], rustc complains that it's referring to a not-in-scope TypeGenerator.

Ideally, TypeGenerator would refer to a full-path variant of itself, to make possible such derives.

That said it's certainly not a big deal, as the fix is simple and quite obvious, I'm just submitting this here as a probably-good-first-issue area of improvement :)

Weird error: Something about `for_each` not being defined a lot of different types.

I'm getting a lot of errors similar to the following:

error[E0599]: no method named `for_each` found for struct `bolero::TestTarget<bolero_generator::TypeValueGenerator<activities::append_only_log::AppendOnlyLog>, bolero::test::TestEngine, bolero::BorrowedInput>` in the current scope
   --> bit_network_library/src/activities/append_only_log.rs:224:46
    |
224 |         fuzz!().with_type::<AppendOnlyLog>().for_each(|_| {
    |                                              ^^^^^^^^ method not found in `bolero::TestTarget<bolero_generator::TypeValueGenerator<activities::append_only_log::AppendOnlyLog>, bolero::test::TestEngine, bolero::BorrowedInput>`
    | 
   ::: /home/cfkaran2/.cargo/registry/src/github.com-1ecc6299db9ec823/bolero-0.5.2/src/lib.rs:149:1
    |
149 | pub struct TestTarget<Generator, Engine, InputOwnership> {
    | -------------------------------------------------------- doesn't satisfy `_: std::iter::Iterator`
    | 
   ::: /home/cfkaran2/.cargo/registry/src/github.com-1ecc6299db9ec823/bolero-0.5.2/src/test/mod.rs:14:1
    |
14  | pub struct TestEngine {
    | --------------------- doesn't satisfy `<_ as bolero_engine::Engine<bolero_engine::test::BorrowedGeneratorTest<_, bolero_generator::TypeValueGenerator<activities::append_only_log::AppendOnlyLog>, _>>>::Output = _`
    |
    = note: the method `for_each` exists but the following trait bounds were not satisfied:
            `<bolero::test::TestEngine as bolero_engine::Engine<bolero_engine::test::BorrowedGeneratorTest<_, bolero_generator::TypeValueGenerator<activities::append_only_log::AppendOnlyLog>, _>>>::Output = _`
            `bolero::TestTarget<bolero_generator::TypeValueGenerator<activities::append_only_log::AppendOnlyLog>, bolero::test::TestEngine, bolero::BorrowedInput>: std::iter::Iterator`
            which is required by `&mut bolero::TestTarget<bolero_generator::TypeValueGenerator<activities::append_only_log::AppendOnlyLog>, bolero::test::TestEngine, bolero::BorrowedInput>: std::iter::Iterator`

the issue is that I'm implemented TypeGenerator on all of these types manually. I'm concerned that they require additional traits implemented on them; do they? If so, which ones?

Meta

lsb_release -a:
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.5 LTS
Release:	18.04
Codename:	bionic

uname -a:
Linux Hammer 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

rustc -vV:
rustc 1.45.2 (d3fb005a3 2020-07-31)
binary: rustc
commit-hash: d3fb005a39e62501b8b0b356166e515ae24e2e54
commit-date: 2020-07-31
host: x86_64-unknown-linux-gnu
release: 1.45.2
LLVM version: 10.0

cargo -vV:
cargo 1.45.1 (f242df6ed 2020-07-22)
release: 1.45.1
commit-hash: f242df6edb897f6f69d393a22bb257f5af0f52d0
commit-date: 2020-07-22

On master, rustc 1.61.0 fails to link fibonacci example with --sanitizer NONE

Hi!

Following #71, I switched to using cargo-bolero from commit 7187c3f.

Then, trying to build the fibonacci example with stable rustc (so 1.61.0 as of this writing), I'm now getting the following linker error:

   Compiling test-bolero v0.1.0 (/home/ekleog/prog/test-bolero)
error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-m64" "/home/ekleog/prog/test-bolero/target/fuzz/build_71e6fd6154e07acb/x86_64-unknown-linux-gnu/release/deps/test_bolero-03796c9c1fd7ff1a.test_bolero.a997a281-cgu.0.rcgu.o" "/home/ekleog/prog/test-bolero/target/fuzz/build_71e6fd6154e07acb/x86_64-unknown-linux-gnu/release/deps/test_bolero-03796c9c1fd7ff1a.rufpas6580vdgtv.rcgu.o" "-Wl,--as-needed" "-L" "/home/ekleog/prog/test-bolero/target/fuzz/build_71e6fd6154e07acb/x86_64-unknown-linux-gnu/release/deps" "-L" "/home/ekleog/prog/test-bolero/target/fuzz/build_71e6fd6154e07acb/release/deps" "-L" "/home/ekleog/prog/test-bolero/target/fuzz/build_71e6fd6154e07acb/x86_64-unknown-linux-gnu/release/build/bolero-libfuzzer-41b6a54ed54d4993/out" "-L" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libtest-efede371ca078f25.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgetopts-84fb18ba7592ffd9.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunicode_width-a1dc360b97cf3afc.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_std-4220d815fd1849e6.rlib" "-Wl,--start-group" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-05b39ac0cb4c5688.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-ccaa149b737d6503.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-a94a6388c153126b.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-1ecfa264246b85c6.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-ee710154a50440b8.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-f09986d873ea8592.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-f3521e20c99619ab.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-df6036c77d75ce21.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-24f91c1a1dfc7e13.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-bffcc61991adb202.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-cf22313353d8663e.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-c45fb100f99430e9.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-0cdc47a8a81950ea.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-f1e158606c4c82f6.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-0b608b5ef4df0ec4.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-b8438dc0bcbbcc08.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-70e2ae036775ff2f.rlib" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-815d281f7068a893.rlib" "-Wl,--end-group" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-377835cfab8dae0d.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/ekleog/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/ekleog/prog/test-bolero/target/fuzz/build_71e6fd6154e07acb/x86_64-unknown-linux-gnu/release/deps/test_bolero-03796c9c1fd7ff1a" "-pie" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs" "-fuse-ld=gold"
  = note: /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/sys_common/backtrace.rs:118: error: undefined reference to '__sancov_lowest_stack'
          /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/function.rs:227: error: undefined reference to '__sanitizer_cov_trace_pc_indir'
          /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/rt.rs:139: error: undefined reference to '__sancov_lowest_stack'
          /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/rt.rs:145: error: undefined reference to '__sancov_lowest_stack'
          /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/fmt/mod.rs:2132: error: undefined reference to '__sancov_lowest_stack'
          /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/cmp.rs:1396: error: undefined reference to '__sanitizer_cov_trace_cmp8'
          /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/fmt/mod.rs:388: error: undefined reference to '__sanitizer_cov_trace_cmp8'
          /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/fmt/mod.rs:388: error: undefined reference to '__sanitizer_cov_trace_cmp8'
          /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/function.rs:227: error: undefined reference to '__sanitizer_cov_trace_pc_indir'
          /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/cmp.rs:1396: error: undefined reference to '__sanitizer_cov_trace_cmp8'
          /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/cmp.rs:1396: error: undefined reference to '__sanitizer_cov_trace_const_cmp8'
          /home/ekleog/prog/test-bolero/target/fuzz/build_71e6fd6154e07acb/x86_64-unknown-linux-gnu/release/deps/test_bolero-03796c9c1fd7ff1a.test_bolero.a997a281-cgu.0.rcgu.o:test_bolero.a997a281-cgu.0:function sancov.module_ctor_8bit_counters: error: undefined reference to '__sanitizer_cov_8bit_counters_init'
          /home/ekleog/prog/test-bolero/target/fuzz/build_71e6fd6154e07acb/x86_64-unknown-linux-gnu/release/deps/test_bolero-03796c9c1fd7ff1a.test_bolero.a997a281-cgu.0.rcgu.o:test_bolero.a997a281-cgu.0:function sancov.module_ctor_8bit_counters: error: undefined reference to '__sanitizer_cov_pcs_init'
          collect2: error: ld returned 1 exit status

  = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

error: could not compile `test-bolero` due to previous error
error: process exited with status 101

It seems to me like libfuzzer is missing some stuff from sancov it's expecting to have when being built with --sanitizer NONE.

When trying with --engine afl, it's only missing __sanitizer_cov_trace_pc_guard, and trying with --engine honggfuzz I'm getting invalid fuzzer "honggfuzz".

Maybe sancov still needs to be enabled even when sanitizers are disabled in order for fuzzers to know the pc trace? But then I'm not sure whether it can actually be enabled on stable :/

`#derive(TypeGenerator)` for recursive `enum`s

Deriving TypeGenerators for recursive types doesn't work. For example:

#[derive(Debug, PartialEq, Eq, Hash, Clone, bolero_generator::TypeGenerator)]
pub enum Expr {
    Int(i32),
    If {
        test_expr: Box<Expr>, // error flagged here on Box constructor
        then_expr: Box<Expr>,
        else_expr: Box<Expr>,
    },
}

and here’s the error

error[E0275]: overflow evaluating the requirement `Expr: TypeGenerator`
  --> src/lib.rs:32:20
   |
32 |         test_expr: Box<Expr>,
   |                    ^^^
   |
   = note: required for `Box<Expr>` to implement `TypeGenerator`
   = help: see issue #48214

It seems cargo fuzz deals with recursive types, but it takes some care.

New generic array impls require higher unwinding bounds for Kani

With the introduction of the impls for TypeGenerator and ValueGenerator in #131, the Kani back-end now requires higher unwinding values (see this change, for example).

We suspect that there's a way to avoid these higher unwinding values somehow, but it's unclear how.

Feature request: Using bolero to find 'slow' inputs

I'm currently using criterion to figure out where my performance bottlenecks are, but it turns out that in some cases my bottleneck is due to the data that is being input. Since this is a data dependency, that suggests that it would be possible to fuzz the input to figure out what inputs are causing the issues. Is it possible to create a new macro similar to check!() that measures performance instead, and emits examples of 'slow' inputs? E.g., perf!(), with the same signature as check!() uses?

`cargo bolero repro` command?

I can't find a way to repro a single fuzzer crash from the cargo bolero command line.

Am I missing something, or should we add such a way, eg. cargo bolero repro [fuzzer] [crash-artifact]?

Currently, the best I found is to use cargo test: it compiles much faster and starts by testing the fuzzer crashes, but even it actually tests all the fuzzer crashes and corpus, and not just the one I'd like to test; which slows down the edit-compile-test cycle

Allow defining the location for corpus and artifacts directories

Hey! I'm thinking of adding an option to cargo bolero test that'd allow setting where one wants the corpus and artifacts directories, as it seems like it already sets that in a way that's independent of the fuzzing engine used. Would you be interested in getting it upstream?

Add support for proptest generators

https://docs.rs/proptest/0.10.1/proptest/strategy/trait.Strategy.html
https://docs.rs/proptest/0.10.1/proptest/arbitrary/trait.Arbitrary.html

Possible APIs:

fn test() {
    bolero::fuzz!()
        .with_proptest(MyStrategy::new())
        .for_each(|_| {
            // assertions here
        })
}

fn test() {
    use proptest::arbitrary::Arbitrary;
    bolero::fuzz!()
        .with_proptest(u64::arbitrary())
        .for_each(|_| {
            // assertions here
        })
}

Add a script for patching integration tests

In #131, we added a CI workflow to check integrations with our dependents. It would be nice to move some of the workflow into a script to make it easier to use.

Global settings for bolero?

Is there a way of setting compile-time settings for all of bolero? I have very complex data structures, which lead to something similar to the following (the real code uses many different collection types, but this was the easiest way to illustrate):

use bolero::generator::*;

#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Hash, TypeGenerator)]
pub struct A {
    field: Vec<u128>
}

#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Hash, TypeGenerator)]
pub struct B {
    field: Vec<A>
}

#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Hash, TypeGenerator)]
pub struct C {
    field: Vec<B>
}

#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Hash, TypeGenerator)]
pub struct D {
    field: Vec<C>
}

#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Hash, TypeGenerator)]
pub struct E {
    field: Vec<D>
}

#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Hash, TypeGenerator)]
pub struct F {
    field: Vec<E>
}

#[cfg(test)]
pub(crate) mod tests {
    use super::*;
    use bolero::fuzz;

    #[test]
    fn new() {
        fuzz!().with_type::<F>().for_each(|_| {
            // Deliberately empty
        });
    }
}

I know I can manually go into each type and annotate the vectors with something like #[generator(Vec::gen().with().len(20usize..50))], but there are a lot of places where I'd need to do this. Is there a simple way of telling bolero that all collections should be limited to 5 items or fewer?

Stats: marking known-useless runs

Assume a fuzzer that looks like:

#[test]
fn fuzzer() {
    check!().for_each(|i: &[u8]| {
        if let Ok(i) = parse_input(i) {
            the_thing_we_actually_want_to_fuzz(i)
        }
    })
}

Basically, any run that does not result in parse_input returning Ok is useless, as the goal of this fuzzer is not semantically to fuzz parse_input (which may be ad-hoc for this fuzzer).

In turn, I'm thinking it might make sense to add a way to report to cargo-bolero that the input was actually (un)interesting, so that it could report stats (eg. "60% of all runs were uninteresting, consider improving parse_int to increase the proportion of valid inputs").

Or even, having the API ending up as something like:

#[test]
fn fuzzer() {
    check!(min_interesting_input_rate=0.6).for_each(|interesting: &mut BoleroMetadata, i: &[u8]| {
        if let Ok(i) = parse_input(i) {
            interesting.is_interesting();
            the_thing_we_actually_want_to_fuzz(i)
        }
    })
}

And then bolero when run in cargo-test mode would run for 1000 times and assert that at least 600 of the runs hit the interesting.is_interesting() line.

This would be bells and whistles, so definitely not essential, but WDYT about it?

cargo-bolero fails to install, hongfuzz libbfd linking error?

Attempting to build using cargo, I found that I was getting linker errors (full text at the bottom). AFAICT, this is a fixed issue in honggfuzz that is discussed here: google/honggfuzz#55 . I am on debian and have binutils-dev and libiberty-dev installed, so I am a bit confused why it is broken for me.

Honggfuzz master branch also compiles without any issue for me. I am guessing that cargo is pulling in some older version of hongfuzz? Is this an issue with bolero, or is it with a dependency?

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.0.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.1.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.10.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.11.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.12.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.13.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.14.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.15.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.2.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.3.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.4.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.5.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.6.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.7.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.8.rcgu.o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.cargo_bolero.6qxf65r1-cgu.9.rcgu.o" "-o" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b" "/tmp/cargo-installbd5wxF/release/deps/cargo_bolero-58b65d9401bc178b.19u6hla6fu586tt5.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/tmp/cargo-installbd5wxF/release/deps" "-L" "/tmp/cargo-installbd5wxF/release/build/bolero-afl-4f11db64a93e9f3b/out" "-L" "/tmp/cargo-installbd5wxF/release/build/bolero-honggfuzz-336c842eafb5a6f1/out" "-L" "/tmp/cargo-installbd5wxF/release/build/backtrace-sys-009cf0958077b0fe/out" "-L" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/tmp/cargo-installbd5wxF/release/deps/libserde_json-e0e70661ee8d30c2.rlib" "/tmp/cargo-installbd5wxF/release/deps/libryu-658c99df18434bd7.rlib" "/tmp/cargo-installbd5wxF/release/deps/libitoa-bf453d7c00eccd10.rlib" "/tmp/cargo-installbd5wxF/release/deps/libtempfile-c93451f56c22e951.rlib" "/tmp/cargo-installbd5wxF/release/deps/librand-67b4b699209d63ec.rlib" "/tmp/cargo-installbd5wxF/release/deps/librand_chacha-bd59041d5b70b99e.rlib" "/tmp/cargo-installbd5wxF/release/deps/libc2_chacha-7bb87f1b9c1e3798.rlib" "/tmp/cargo-installbd5wxF/release/deps/libppv_lite86-326b87106567ac09.rlib" "/tmp/cargo-installbd5wxF/release/deps/librand_core-437e13fc8c4ea9d5.rlib" "/tmp/cargo-installbd5wxF/release/deps/libgetrandom-2637594b360181ee.rlib" "/tmp/cargo-installbd5wxF/release/deps/libremove_dir_all-49d5c8d2ff8c14fe.rlib" "/tmp/cargo-installbd5wxF/release/deps/libbolero_honggfuzz-af537a38b9d9076b.rlib" "/tmp/cargo-installbd5wxF/release/deps/libbolero_afl-e7d437144aec9a8c.rlib" "/tmp/cargo-installbd5wxF/release/deps/libserde-5d3c50cde77ed1aa.rlib" "/tmp/cargo-installbd5wxF/release/deps/libstructopt-24891eefa8cd090c.rlib" "/tmp/cargo-installbd5wxF/release/deps/libclap-84f2b70592138cc2.rlib" "/tmp/cargo-installbd5wxF/release/deps/libvec_map-52e45a5cc4c7f387.rlib" "/tmp/cargo-installbd5wxF/release/deps/libtextwrap-3f16ff81e65c9478.rlib" "/tmp/cargo-installbd5wxF/release/deps/libunicode_width-54e6d2665cd8f093.rlib" "/tmp/cargo-installbd5wxF/release/deps/libstrsim-80455a6b3d8c706d.rlib" "/tmp/cargo-installbd5wxF/release/deps/libbitflags-34d6e20d451463ab.rlib" "/tmp/cargo-installbd5wxF/release/deps/libatty-e7a88ecbc75770fc.rlib" "/tmp/cargo-installbd5wxF/release/deps/libansi_term-24e088762a72cae0.rlib" "/tmp/cargo-installbd5wxF/release/deps/libfailure-4843a47ae8c6b2f5.rlib" "/tmp/cargo-installbd5wxF/release/deps/libbacktrace-e16d357a2b2fd27c.rlib" "/tmp/cargo-installbd5wxF/release/deps/libbacktrace_sys-36f714458bea0103.rlib" "/tmp/cargo-installbd5wxF/release/deps/liblibc-3a23dd11839429e0.rlib" "/tmp/cargo-installbd5wxF/release/deps/libcfg_if-f3544b1da576ae5e.rlib" "/tmp/cargo-installbd5wxF/release/deps/librustc_demangle-bffb1c1505d17a08.rlib" "-Wl,--start-group" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-74488c47a41eb313.rlib" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-efbc2c947951b8cc.rlib" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-b177382d9a04ffbe.rlib" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-ba7d2244d33447d5.rlib" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace-39569dc87e4ea301.rlib" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-cd9e255bd82ddb5c.rlib" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-f83165e777dccb8d.rlib" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-10b085de263b1750.rlib" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-9a4a22edf44da957.rlib" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-68fabc677efa98de.rlib" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-026dc0061b48e8b6.rlib" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-7bf75bb619341145.rlib" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-ef54709e300503ed.rlib" "-Wl,--end-group" "/home/jeremysalwen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-4ed27d8420cb4abc.rlib" "-Wl,-Bdynamic" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
  = note: /usr/bin/ld: /tmp/cargo-installbd5wxF/release/deps/libbolero_honggfuzz-af537a38b9d9076b.rlib(unwind.o): in function `arch_unwindStack':
          unwind.c:(.text+0x129): undefined reference to `_UPT_accessors'
          /usr/bin/ld: unwind.c:(.text+0x12e): undefined reference to `_Ux86_64_create_addr_space'
          /usr/bin/ld: unwind.c:(.text+0x145): undefined reference to `_UPT_create'
          /usr/bin/ld: unwind.c:(.text+0x16b): undefined reference to `_Ux86_64_init_remote'
          /usr/bin/ld: unwind.c:(.text+0x1b2): undefined reference to `_Ux86_64_step'
          /usr/bin/ld: unwind.c:(.text+0x1d4): undefined reference to `_Ux86_64_get_reg'
          /usr/bin/ld: unwind.c:(.text+0x48d): undefined reference to `_Ux86_64_destroy_addr_space'
          /usr/bin/ld: unwind.c:(.text+0x55e): undefined reference to `_UPT_destroy'
          /usr/bin/ld: /tmp/cargo-installbd5wxF/release/deps/libbolero_honggfuzz-af537a38b9d9076b.rlib(bfd.o): in function `arch_bfdInit.constprop.3':
          bfd.c:(.text+0xcd): undefined reference to `bfd_openr'
          /usr/bin/ld: bfd.c:(.text+0xe6): undefined reference to `bfd_check_format'
          /usr/bin/ld: bfd.c:(.text+0x12a): undefined reference to `bfd_get_section_by_name'
          /usr/bin/ld: /tmp/cargo-installbd5wxF/release/deps/libbolero_honggfuzz-af537a38b9d9076b.rlib(bfd.o): in function `arch_bfdResolveSyms':
          bfd.c:(.text+0x28f): undefined reference to `bfd_init'
          /usr/bin/ld: bfd.c:(.text+0x3c2): undefined reference to `bfd_close'
          /usr/bin/ld: /tmp/cargo-installbd5wxF/release/deps/libbolero_honggfuzz-af537a38b9d9076b.rlib(bfd.o): in function `arch_bfdDisasm':
          bfd.c:(.text+0x42d): undefined reference to `bfd_init'
          /usr/bin/ld: bfd.c:(.text+0x44f): undefined reference to `bfd_openr'
          /usr/bin/ld: bfd.c:(.text+0x468): undefined reference to `bfd_check_format'
          /usr/bin/ld: bfd.c:(.text+0x49c): undefined reference to `bfd_close'
          /usr/bin/ld: bfd.c:(.text+0x4e0): undefined reference to `bfd_get_arch'
          /usr/bin/ld: bfd.c:(.text+0x4ef): undefined reference to `disassembler'
          /usr/bin/ld: bfd.c:(.text+0x510): undefined reference to `init_disassemble_info'
          /usr/bin/ld: bfd.c:(.text+0x518): undefined reference to `bfd_get_arch'
          /usr/bin/ld: bfd.c:(.text+0x524): undefined reference to `bfd_get_mach'
          /usr/bin/ld: bfd.c:(.text+0x55c): undefined reference to `disassemble_init_for_target'
          /usr/bin/ld: bfd.c:(.text+0x575): undefined reference to `bfd_close'
          collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

error: failed to compile `cargo-bolero v0.3.0`, intermediate artifacts can be found at `/tmp/cargo-installbd5wxF`

How to write generic tests?

I've found myself writing a lot of code that is very repetitious while writing tests. Basically, I'm trying to take something like the following:

#[cfg(test)]
mod tests {
    use bolero::fuzz;
    use std::{
        collections::hash_map::DefaultHasher,
        hash::{Hash, Hasher},
    };

    fn calculate_hash<T: Hash>(t: &T) -> u64 {
        let mut s = DefaultHasher::new();
        t.hash(&mut s);
        s.finish()
    }

    #[test]
    fn test() {
        fuzz!().with_type::<(u32, u32)>().for_each(|(left, right)| {
            if left == right {
                assert_eq!(calculate_hash(left), calculate_hash(right));
            }
        });
    }
}

and turn it into something like:

#[cfg(test)]
mod tests {
    use bolero::{fuzz, generator::*};
    use std::{
        cmp::Eq,
        collections::hash_map::DefaultHasher,
        fmt::Debug,
        hash::{Hash, Hasher},
    };

    fn calculate_hash<T: Hash>(t: &T) -> u64 {
        let mut s = DefaultHasher::new();
        t.hash(&mut s);
        s.finish()
    }

    fn test_hash<T>()
    where
        T: Debug + TypeGenerator + Hash + Eq,
    {
        fuzz!().with_type::<(T, T)>().for_each(|(left, right)| {
            if left == right {
                assert_eq!(calculate_hash(left), calculate_hash(right));
            }
        });
    }

    #[test]
    fn test() {
        test_hash::<u32>();
    }
}

with the goal of defining all of my test functions (like fn test_hash<T>()) in a single crate that can then be shared everywhere.

The problem is that the second chunk of code yields the following error on compilation:

error[E0599]: no method named `for_each` found for struct `bolero::TestTarget<bolero_generator::TypeValueGenerator<(T, T)>, bolero::test::TestEngine, bolero::BorrowedInput>` in the current scope
   --> src/main.rs:25:39
    |
25  |         fuzz!().with_type::<(T, T)>().for_each(|(left, right)| {
    |                                       ^^^^^^^^ method not found in `bolero::TestTarget<bolero_generator::TypeValueGenerator<(T, T)>, bolero::test::TestEngine, bolero::BorrowedInput>`
    | 
   ::: /home/cfkaran2/.cargo/registry/src/github.com-1ecc6299db9ec823/bolero-0.5.2/src/lib.rs:149:1
    |
149 | pub struct TestTarget<Generator, Engine, InputOwnership> {
    | -------------------------------------------------------- doesn't satisfy `_: std::iter::Iterator`
    | 
   ::: /home/cfkaran2/.cargo/registry/src/github.com-1ecc6299db9ec823/bolero-0.5.2/src/test/mod.rs:14:1
    |
14  | pub struct TestEngine {
    | --------------------- doesn't satisfy `<_ as bolero_engine::Engine<bolero_engine::test::BorrowedGeneratorTest<_, bolero_generator::TypeValueGenerator<(T, T)>, _>>>::Output = _`
    |
    = note: the method `for_each` exists but the following trait bounds were not satisfied:
            `<bolero::test::TestEngine as bolero_engine::Engine<bolero_engine::test::BorrowedGeneratorTest<_, bolero_generator::TypeValueGenerator<(T, T)>, _>>>::Output = _`
            `bolero::TestTarget<bolero_generator::TypeValueGenerator<(T, T)>, bolero::test::TestEngine, bolero::BorrowedInput>: std::iter::Iterator`
            which is required by `&mut bolero::TestTarget<bolero_generator::TypeValueGenerator<(T, T)>, bolero::test::TestEngine, bolero::BorrowedInput>: std::iter::Iterator`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0599`.
error: could not compile `trait_test`.

To learn more, run the command again with --verbose.

I know that the issue is that I don't have the right trait bounds in place, but I'm not sure what is expected here. Do you have any suggestions?

Detangling Driver and DriverMode

So, continuing the discussion started on #108 (comment) ; let's chat about driver and driver mode here.

I guess my first question would be, is there any user-facing use case for something other than (DriverMode::Forced, Driver::Rng) (for proptesting) and (DriverMode::Direct, Driver::ByteSlice) (for fuzzers)? And is there a non-user-facing use case for anything other than (DriverMode::Forced, Driver::ByteSlice), that's I guess used for input shrinking when proptesting?

In particular, do you know why DirectRng is being exposed?

Also, writing down my current plan for changes of *Generator while I have it in mind: instead of fn(Driver) -> Generated, have it be fn(Driver, DriverMode, depth: usize) -> Generated. Use the depth to generate smaller collections when going deeper in Forced mode, and to refuse generating too deep data structures like arbitrary in Direct mode.

Running with -j 8 leads to "error: Unrecognized option: 'a'"

Oops, I got tricked once again by github by submitting early.

If I use bolero 0.8.0 and cargo-bolero 0.8.0, and attempt to run a fuzzer with -j 8, hoping to see 8 jobs, instead I see this result:

cargo bolero test tests::succeed -j 8
    Finished release [optimized] target(s) in 0.02s
  Executable unittests src/lib.rs (target/fuzz/build_58d39c7ddcef2bdf/x86_64-unknown-linux-gnu/release/deps/foo-b9bbcb1c97adc7ba)

running 1 test
Running 4 workers
/tmp/foo/target/fuzz/build_58d39c7ddcef2bdf/x86_64-unknown-linux-gnu/release/deps/foo-b9bbcb1c97adc7ba /tmp/foo/src/__fuzz__/tests__succeed/corpus /tmp/foo/src/__fuzz__/tests__succeed/crashes -artifact_prefix=/tmp/foo/src/__fuzz__/tests__succeed/crashes/ -timeout=10 >fuzz-0.log 2>&1
/tmp/foo/target/fuzz/build_58d39c7ddcef2bdf/x86_64-unknown-linux-gnu/release/deps/foo-b9bbcb1c97adc7ba /tmp/foo/src/__fuzz__/tests__succeed/corpus /tmp/foo/src/__fuzz__/tests__succeed/crashes -artifact_prefix=/tmp/foo/src/__fuzz__/tests__succeed/crashes/ -timeout=10 >fuzz-1.log 2>&1
/tmp/foo/target/fuzz/build_58d39c7ddcef2bdf/x86_64-unknown-linux-gnu/release/deps/foo-b9bbcb1c97adc7ba /tmp/foo/src/__fuzz__/tests__succeed/corpus /tmp/foo/src/__fuzz__/tests__succeed/crashes -artifact_prefix=/tmp/foo/src/__fuzz__/tests__succeed/crashes/ -timeout=10 >fuzz-2.log 2>&1
/tmp/foo/target/fuzz/build_58d39c7ddcef2bdf/x86_64-unknown-linux-gnu/release/deps/foo-b9bbcb1c97adc7ba /tmp/foo/src/__fuzz__/tests__succeed/corpus /tmp/foo/src/__fuzz__/tests__succeed/crashes -artifact_prefix=/tmp/foo/src/__fuzz__/tests__succeed/crashes/ -timeout=10 >fuzz-3.log 2>&1
================== Job 1 exited with exit code 101 ============
error: Unrecognized option: 'a'
================== Job 0 exited with exit code 101 ============
/tmp/foo/target/fuzz/build_58d39c7ddcef2bdf/x86_64-unknown-linux-gnu/release/deps/foo-b9bbcb1c97adc7ba /tmp/foo/src/__fuzz__/tests__succeed/corpus /tmp/foo/src/__fuzz__/tests__succeed/crashes -artifact_prefix=/tmp/foo/src/__fuzz__/tests__succeed/crashes/ -timeout=10 >fuzz-4.log 2>&1
error: Unrecognized option: 'a'
/tmp/foo/target/fuzz/build_58d39c7ddcef2bdf/x86_64-unknown-linux-gnu/release/deps/foo-b9bbcb1c97adc7ba /tmp/foo/src/__fuzz__/tests__succeed/corpus /tmp/foo/src/__fuzz__/tests__succeed/crashes -artifact_prefix=/tmp/foo/src/__fuzz__/tests__succeed/crashes/ -timeout=10 >fuzz-5.log 2>&1
================== Job 2 exited with exit code 101 ============
error: Unrecognized option: 'a'
/tmp/foo/target/fuzz/build_58d39c7ddcef2bdf/x86_64-unknown-linux-gnu/release/deps/foo-b9bbcb1c97adc7ba /tmp/foo/src/__fuzz__/tests__succeed/corpus /tmp/foo/src/__fuzz__/tests__succeed/crashes -artifact_prefix=/tmp/foo/src/__fuzz__/tests__succeed/crashes/ -timeout=10 >fuzz-6.log 2>&1
================== Job 3 exited with exit code 101 ============
error: Unrecognized option: 'a'
/tmp/foo/target/fuzz/build_58d39c7ddcef2bdf/x86_64-unknown-linux-gnu/release/deps/foo-b9bbcb1c97adc7ba /tmp/foo/src/__fuzz__/tests__succeed/corpus /tmp/foo/src/__fuzz__/tests__succeed/crashes -artifact_prefix=/tmp/foo/src/__fuzz__/tests__succeed/crashes/ -timeout=10 >fuzz-7.log 2>&1
================== Job 4 exited with exit code 101 ============
error: Unrecognized option: 'a'
================== Job 5 exited with exit code 101 ============
error: Unrecognized option: 'a'
================== Job 6 exited with exit code 101 ============
error: Unrecognized option: 'a'
================== Job 7 exited with exit code 101 ============
error: Unrecognized option: 'a'

My guess is, it’s because it’s passing the libfuzzer arguments directly to the unit tests binary instead of through BOLERO_LIBFUZZER_ARGS.

Code examples of using bolero-generator

The readme shows examples of basic unit test, but doesn't show how bolero_generator api can be used to restrict values to a range. The bolero_generator crate is linked to, but the relationship between the two apis is unclear, because the examples all use it implicitly.

Even the cargo docs only show a single example of generators being explicitly created: https://docs.rs/bolero/0.4.1/bolero/macro.check.html and doesn't demonstrate any of the features of the api demonstrated here: https://crates.io/crates/bolero-generator so it was easy to miss.

Add support for Arbitrary generators

https://docs.rs/arbitrary/0.4.7/arbitrary/trait.Arbitrary.html

Possible APIs:

fn test() {
    bolero::fuzz!()
        .with_arbitrary::<u64>()
        .for_each(|_| {
            // assertions here
        })
}

Compiling when specifying honggfuzz as fuzzer fails.

Context

OS: Ubuntu 20.04 LTS
Rust version:

stable-x86_64-unknown-linux-gnu (default)
rustc 1.47.0 (18bf6b4f0 2020-10-07)

Dependencies:

├── bolero v0.5.2
│   ├── bolero-engine v0.5.2
│   │   ├── bolero-generator v0.5.2
│   │   │   ├── bolero-generator-derive v0.5.2
│   ├── bolero-generator v0.5.2 (*)

Command:

cargo bolero fuzz streaming --sanitizer address --fuzzer honggfuzz

Error output:

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-Wl,--eh-frame-hdr" "-Wl,-Bstatic" "-Wl,--whole-archive" <many many .a/.rlib paths> "-Wl,-Bdynamic" "-lutil" "-ldl" "-lutil" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-lhfuzz" "-fuse-ld=gold"
  = note: libhfuzz/instrument.c:230: error: undefined reference to 'logLog'
          libhfuzz/instrument.c:396: error: undefined reference to 'util_32bitValInBinary'
          libhfuzz/instrument.c:389: error: undefined reference to 'util_32bitValInBinary'
          libhfuzz/instrument.c:389: error: undefined reference to 'util_32bitValInBinary'
          libhfuzz/instrument.c:396: error: undefined reference to 'util_32bitValInBinary'
          libhfuzz/instrument.c:418: error: undefined reference to 'util_64bitValInBinary'
          libhfuzz/instrument.c:411: error: undefined reference to 'util_64bitValInBinary'
          libhfuzz/instrument.c:411: error: undefined reference to 'util_64bitValInBinary'
          libhfuzz/instrument.c:418: error: undefined reference to 'util_64bitValInBinary'
          libhfuzz/instrument.c:599: error: undefined reference to 'hf_log_level'
          libhfuzz/instrument.c:600: error: undefined reference to 'logLog'
          libhfuzz/instrument.c:604: error: undefined reference to 'logLog'
          libhfuzz/instrument.c:754: error: undefined reference to 'hf_log_level'
          libhfuzz/instrument.c:754: error: undefined reference to 'logLog'
          libhfuzz/instrument.c:187: error: undefined reference to 'hf_log_level'
          libhfuzz/instrument.c:99: error: undefined reference to 'files_getTmpMapFlags'
          libhfuzz/instrument.c:185: error: undefined reference to 'logInitLogFile'
          libhfuzz/instrument.c:187: error: undefined reference to 'hf_log_level'
          libhfuzz/instrument.c:111: error: undefined reference to 'files_getTmpMapFlags'
          libhfuzz/instrument.c:99: error: undefined reference to 'files_getTmpMapFlags'
          libhfuzz/instrument.c:99: error: undefined reference to 'files_getTmpMapFlags'
          libhfuzz/instrument.c:812: error: undefined reference to 'util_getProgAddr'
          libhfuzz/instrument.c:825: error: undefined reference to 'util_getProgAddr'
          libhfuzz/instrument.c:841: error: undefined reference to 'util_getProgAddr'
          collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

Add support for Insta assertions

https://docs.rs/insta/1.1.0/insta/

fn test() {
    bolero::fuzz!()
        .with_type()
        .for_each(|value: u64| {
            // would only be compared if we're running a corpus replay or possibly finishing a
            // fuzz/reduce command

            // it should also have the same/similar name as the corpus file
            bolero::assert_debug_snapshot!(my_function(value));
        })
}

Feature request: using bolero in other languages

This is a very long-term feature request, please put it down as a low priority.

Bolero is my go-to tool for testing all of my code in rust. Unfortunately, I can't use rust everywhere; in particular, I have to use C++ at my day job. Is there any way to make bolero work with C/C++ with a minimum of effort? If not, I'll spend the time figuring out FFI bindings (if you think that will work).

Running cargo-bolero jobs on ClusterFuzz

Hey! I’m currently looking into both cargo-bolero and ClusterFuzz (the software under oss-fuzz).

I’m curious, has anyone run a cargo-bolero-built fuzzer on top of ClusterFuzz yet? It seems to me like ClusterFuzz needs a libfuzzer binary with the CLI of a regular libfuzzer binary to properly run.

My first thoughts were that I could try patching ClusterFuzz to also support bolero’s CLI (patching around here and other run_and_wait calls), but seeing how ClusterFuzz is google-owned I’m not sure this would be the best way forward, especially as cargo-bolero is not yet that widespread.

My current thoughts are, that we could add a command to cargo-bolero to generate a clusterfuzz tgz for a given cargo-bolero target. This clusterfuzz tgz would contain the regular cargo-bolero binary, but the actual binary for clusterfuzz (so, the fuzzer binary) would be a mini-binary that just takes its arguments, puts them into BOLERO_LIBFUZZER_ARGS, and calls the bolero binary with as argument the test to run and this environment variable.

Does that make sense as a plan to you?

Can you run multiple tests in parallel?

I'm using GNU parallel to parallelize all testing under bolero; I basically used cargo bolero list > all_tests, and then then hacked the result to look similar to the following:

cargo bolero fuzz --time 60sec activities::node_activities::error::tests::clone
cargo bolero fuzz --time 60sec activities::node_activities::error::tests::debug
cargo bolero fuzz --time 60sec activities::node_activities::error::tests::partial_eq
cargo bolero fuzz --time 60sec activities::node_activities::error::tests::partial_ord
cargo bolero fuzz --time 60sec activities::node_activities::error::tests::serialize
...

I then run everything using

cat all_tests | parallel --progress --eta --nice 10

which seems to work, but it occurred to me that bolero might not be able to correctly handle being run on the same project concurrently. So, is it safe to do this? What about if I remove the target directory in between, forcing bolero to rebuild when the command is run?

`reduce` command causes tool to crash

Wrote a simple bolero test like this one:

#[test]
fn function() {
    check!()
        .with_type::<i32>()
        .for_each(|x| {
            assert!(*x < 100_i32);
        })
}

When I run cargo bolero reduce function it crashes:

running 1 test
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1694551407
INFO: Loaded 1 modules   (294218 inline 8-bit counters): 294218 [0x107c22948, 0x107c6a692), 
INFO: Loaded 1 PC tables (294218 PCs): 294218 [0x107c6a698,0x1080e7b38), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/var/folders/qb/tckm4wwd0gzbl9ypc53ncdb8xg9k_h/T/.tmpoaEcQo'
MERGE: failed to parse the control file (unexpected error)
error: process exited with status 1

Looks like it's failing to parse a file.

Building from source fails

I'm unable to build bolero from source:

$ git clone [email protected]:camshaft/bolero.git
Cloning into 'bolero'...
remote: Enumerating objects: 2526, done.
remote: Counting objects: 100% (237/237), done.
remote: Compressing objects: 100% (94/94), done.
remote: Total 2526 (delta 158), reused 168 (delta 139), pack-reused 2289
Receiving objects: 100% (2526/2526), 2.06 MiB | 12.46 MiB/s, done.
Resolving deltas: 100% (1385/1385), done.
$ cd bolero/
$ cargo build
    Updating crates.io index
error: package `os_str_bytes v6.2.0` cannot be built because it requires rustc 1.57.0 or newer, while the currently active rustc version is 1.56.1

The os_str_bytes crate is an indirect dependency of libtest-mimic:

$ cargo tree -i os_str_bytes
os_str_bytes v6.2.0
└── clap_lex v0.2.4
    └── clap v3.2.16
        └── libtest-mimic v0.4.1
            └── bolero v0.6.2 (/home/ubuntu/git/tmptmp/bolero/bolero)
                [dev-dependencies]
                └── cargo-bolero v0.6.2 (/home/ubuntu/git/tmptmp/bolero/cargo-bolero)

which was bumped to version 0.4.1 recently via 5578547.

OS version: Ubuntu 20.04

Possible bug in check!()

I'm using check!() in my unit tests, which calls out to an implementation of TypeGenerator. The implementation is below (I've tried to remove as much stuff as possible for clarity, so the code won't compile as it is). If you read through the implementation for TypeGenerator, you'll see that it is generating a semver 2.0 compliant value, which is then stuffed into a semver::Version instance. The problem is the loop in gen_alpha_numeric_ascii_character(); it assumes that either the driver will return None, which will then get propagated all the way back up, or it will receive a new, randomly generated value. Given enough time, that would guarantee that the loop would exit. However, in testing I've discovered that the generator will eventually start returning 0 rather than run out of values. That means I get stuck in here. I've done a temporary workaround where I instead of the loop, I use an ordinary for loop and loop 100 times, returning None if I reach the end of the loop. But this isn't quite what I'd expect out of this.

use semver::Version;

pub struct Meta {
    /// [SemVer 2.0](https://semver.org/spec/v2.0.0.html) compliant value.
    pub version: Version,
}

impl TypeGenerator for Meta {
    fn generate<R>(driver: &mut R) -> Option<Self>
    where
        R: Driver,
    {
        fn gen_alpha_numeric_ascii_character<R>(driver: &mut R) -> Option<char>
        where
            R: Driver,
        {
            let mut value: u8 = driver.gen()?;
            loop {
                if (value >= 0x41 && value <= 0x5A)
                    || (value >= 0x61 && value <= 0x7A)
                    || (value >= 0x30 && value <= 0x39)
                {
                    break;
                } else {
                    value = driver.gen()?;
                }
            }

            Some(char::from(value))
        }

        fn gen_alpha_numeric_string<R>(driver: &mut R) -> Option<String>
        where
            R: Driver,
        {
            let len: usize = driver.gen()?;

            // This is an arbitrary limit to the number of characters.  If this
            // is too small, increase it as needed.
            let len = len % 20;

            let mut output: String = String::new();
            for _ in 0..len {
                let value = gen_alpha_numeric_ascii_character(driver)?;
                output.push(value);
            }

            Some(output)
        }

        fn gen_identifier<R>(driver: &mut R) -> Option<Identifier>
        where
            R: Driver,
        {
            if driver.gen()? {
                Some(Identifier::Numeric(driver.gen()?))
            } else {
                let num: usize = driver.gen()?;
                let num = num % 5;
                let mut parts: Vec<String> = Vec::new();
                for _ in 0..num {
                    let alpha_num_string = gen_alpha_numeric_string(driver)?;
                    parts.push(alpha_num_string);
                }
                let output: String = join(&parts, ".");

                Some(Identifier::AlphaNumeric(output))
            }
        }

        fn gen_vec_of_identifiers<R>(driver: &mut R) -> Option<Vec<Identifier>>
        where
            R: Driver,
        {
            let num: usize = driver.gen()?;
            let num = num % 5;
            let mut output: Vec<Identifier> = Vec::new();
            for _ in 0..num {
                let identifier = gen_identifier(driver)?;
                output.push(identifier);
            }

            Some(output)
        }

        let major: u64 = driver.gen()?;
        let minor: u64 = driver.gen()?;
        let patch: u64 = driver.gen()?;
        let pre: Vec<Identifier> = gen_vec_of_identifiers(driver)?;
        let build: Vec<Identifier> = gen_vec_of_identifiers(driver)?;

        let version: Version = Version {
            major,
            minor,
            patch,
            pre,
            build,
        };

        Some(Meta {
            version,
        })
    }
}

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: UnexpectedVersionFormat'

While trying to run bolero following the instructions from the README.md, I ran into this error:

$ cargo bolero test fuzz_add
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: UnexpectedVersionFormat', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/cargo-bolero-0.8.0/src/project.rs:167:57
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

$ env RUST_BACKTRACE=1 cargo bolero test fuzz_add
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: UnexpectedVersionFormat', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/cargo-bolero-0.8.0/src/project.rs:167:57
stack backtrace:
   0: rust_begin_unwind
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panicking.rs:142:14
   2: core::result::unwrap_failed
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/result.rs:1785:5
   3: cargo_bolero::project::Project::rustflags
   4: cargo_bolero::project::Project::cmd
   5: cargo_bolero::selection::Selection::test_target
   6: cargo_bolero::libfuzzer::test
   7: cargo_bolero::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Any ideas?

Fibonacci example makes stable rustc 1.60 segfault in LLVM

Hello!

So first of all, I recently learned of cargo-bolero, and it looks like an awesome piece of software, much neater to use than cargo-fuzz!

That said, I'm having trouble running the fibonacci example on rustc stable.

Here is my code from following the tutorial until the first cargo bolero test command: https://github.com/Ekleog/test-bolero/tree/main/tests/something

I have just added a bolero = "0.6" dependency to Cargo.toml that seemed required, in addition to following the tutorial.

However, if I try running cargo bolero test something --sanitizer NONE, it's crashing rustc as a segfault inside LLVM:

   Compiling byteorder v1.4.3
   Compiling backtrace v0.3.65
   Compiling bolero-generator v0.6.2
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x4b3d23)[0x7fc860fafd23]
/nix/store/hbvq1160w7k0k0cfpmpd1hn3gf6apika-glibc-2.34-115/lib/libc.so.6(+0x41100)[0x7fc860581100]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(+0x30993c4)[0x7fc85e18b3c4]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(_ZN15LiveDebugValues16InstrRefBasedLDV21depthFirstVLocAndEmitEjRKN4llvm8DenseMapIPKNS1_12LexicalScopeEPKNS1_10DILocationENS1_12DenseMapInfoIS5_vEENS1_6detail12DenseMapPairIS5_S8_EEEERKNS2_IS5_NS1_8SmallSetINS1_13DebugVariableELj4ESt4lessISI_EEESA_NSC_IS5_SL_EEEERNS2_IS5_NS1_11SmallPtrSetIPNS1_17MachineBasicBlockELj4EEESA_NSC_IS5_ST_EEEERNS1_11SmallVectorINSX_ISt4pairISI_NS_8DbgValueEELj8EEELj8EEEPPNS_10ValueIDNumES16_RNS1_15SmallVectorImplINS_11VLocTrackerEEERNS1_15MachineFunctionERNS2_ISI_jNS9_ISI_vEENSC_ISI_jEEEERKNS1_16TargetPassConfigE+0x691)[0x7fc85e18a661]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(_ZN15LiveDebugValues16InstrRefBasedLDV12ExtendRangesERN4llvm15MachineFunctionEPNS1_20MachineDominatorTreeEPNS1_16TargetPassConfigEjj+0xc2b)[0x7fc85e1878bb]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE+0xaa)[0x7fc85e0b8e6a]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE+0x6b2)[0x7fc85de6bab2]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE+0x2f)[0x7fc85de6b3df]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE+0x289)[0x7fc85e36c439]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x23dc954)[0x7fc862ed8954]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x23cbf7d)[0x7fc862ec7f7d]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x23cf05c)[0x7fc862ecb05c]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x2366c28)[0x7fc862e62c28]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x2365e22)[0x7fc862e61e22]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x238380f)[0x7fc862e7f80f]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x2372bc7)[0x7fc862e6ebc7]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/libstd-8d61b92a0a02f53a.so(rust_metadata_std_cd3cf6af28dff6de+0xa7d03)[0x7fc8607f7d03]
/nix/store/hbvq1160w7k0k0cfpmpd1hn3gf6apika-glibc-2.34-115/lib/libc.so.6(+0x89eb2)[0x7fc8605c9eb2]
/nix/store/hbvq1160w7k0k0cfpmpd1hn3gf6apika-glibc-2.34-115/lib/libc.so.6(+0x10c31c)[0x7fc86064c31c]
error: could not compile `byteorder`

Caused by:
  process didn't exit successfully: `rustc --crate-name byteorder --edition=2018 /home/ekleog/.cargo/registry/src/github.com-1ecc6299db9ec823/byteorder-1.4.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=adfaf6a15e611a6e -C extra-filename=-adfaf6a15e611a6e --out-dir /home/ekleog/prog/test-bolero/target/fuzz/build_70eaf36aa76f6a54/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/home/ekleog/prog/test-bolero/target/fuzz/build_70eaf36aa76f6a54/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/ekleog/prog/test-bolero/target/fuzz/build_70eaf36aa76f6a54/release/deps --cap-lints allow --cfg fuzzing -Cdebug-assertions -Ctarget-cpu=native -Cdebuginfo=2 -Coverflow_checks -Clink-dead-code -Cpasses=sancov-module --cfg fuzzing_libfuzzer -Cllvm-args=-sanitizer-coverage-inline-8bit-counters -Cllvm-args=-sanitizer-coverage-level=4 -Cllvm-args=-sanitizer-coverage-pc-table -Cllvm-args=-sanitizer-coverage-prune-blocks=0 -Cllvm-args=-sanitizer-coverage-trace-compares -Cllvm-args=-sanitizer-coverage-trace-divs -Cllvm-args=-sanitizer-coverage-trace-geps -Cllvm-args=-sanitizer-coverage-stack-depth -Clink-arg=-fuse-ld=gold -Ccodegen-units=1` (signal: 11, SIGSEGV: invalid memory reference)
warning: build failed, waiting for other jobs to finish...
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x4b3d23)[0x7f9e5a834d23]
/nix/store/hbvq1160w7k0k0cfpmpd1hn3gf6apika-glibc-2.34-115/lib/libc.so.6(+0x41100)[0x7f9e59e06100]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(+0x30993c4)[0x7f9e57a103c4]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(_ZN15LiveDebugValues16InstrRefBasedLDV21depthFirstVLocAndEmitEjRKN4llvm8DenseMapIPKNS1_12LexicalScopeEPKNS1_10DILocationENS1_12DenseMapInfoIS5_vEENS1_6detail12DenseMapPairIS5_S8_EEEERKNS2_IS5_NS1_8SmallSetINS1_13DebugVariableELj4ESt4lessISI_EEESA_NSC_IS5_SL_EEEERNS2_IS5_NS1_11SmallPtrSetIPNS1_17MachineBasicBlockELj4EEESA_NSC_IS5_ST_EEEERNS1_11SmallVectorINSX_ISt4pairISI_NS_8DbgValueEELj8EEELj8EEEPPNS_10ValueIDNumES16_RNS1_15SmallVectorImplINS_11VLocTrackerEEERNS1_15MachineFunctionERNS2_ISI_jNS9_ISI_vEENSC_ISI_jEEEERKNS1_16TargetPassConfigE+0x691)[0x7f9e57a0f661]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(_ZN15LiveDebugValues16InstrRefBasedLDV12ExtendRangesERN4llvm15MachineFunctionEPNS1_20MachineDominatorTreeEPNS1_16TargetPassConfigEjj+0xc2b)[0x7f9e57a0c8bb]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE+0xaa)[0x7f9e5793de6a]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE+0x6b2)[0x7f9e576f0ab2]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE+0x2f)[0x7f9e576f03df]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.60.0-stable.so(_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE+0x289)[0x7f9e57bf1439]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x23dc954)[0x7f9e5c75d954]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x23cbf7d)[0x7f9e5c74cf7d]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x23cf05c)[0x7f9e5c75005c]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x2366c28)[0x7f9e5c6e7c28]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x2365e22)[0x7f9e5c6e6e22]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x238380f)[0x7f9e5c70480f]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-75e5f32fc3580f6c.so(+0x2372bc7)[0x7f9e5c6f3bc7]
/home/ekleog/.rustup/toolchains/1.60.0-x86_64-unknown-linux-gnu/bin/../lib/libstd-8d61b92a0a02f53a.so(rust_metadata_std_cd3cf6af28dff6de+0xa7d03)[0x7f9e5a07cd03]
/nix/store/hbvq1160w7k0k0cfpmpd1hn3gf6apika-glibc-2.34-115/lib/libc.so.6(+0x89eb2)[0x7f9e59e4eeb2]
/nix/store/hbvq1160w7k0k0cfpmpd1hn3gf6apika-glibc-2.34-115/lib/libc.so.6(+0x10c31c)[0x7f9e59ed131c]
error: build failed
error: process exited with status 101

Is this something you've already seen happening, and/or reported upstream to rustc? Do you have any idea where I should be looking around to try and debug this?
Also, interestingly running with nightly rust (so without --sanitizer NONE and with installing a nightly toolchain), it's no longer failing.

I'd love to switch our use of cargo-fuzz to using cargo-bolero instead, but if it requires a nightly toolchain to be installed it'd be much harder unfortunately (we're using cargo-fuzz with RUSTC_BOOTSTRAP currently, but it looks like cargo-bolero doesn't proxy this trick)

New release?

Now that #18 is fixed, maybe it's time for a new release?

Make a release?

Right now the latest (0.7.0) release of bolero depends on libtest-mimic 0.3, which depends on crossbeam-channel 0.4, which has a security vulnerability.

It's certainly not a big deal as bolero is only used for tests anyway, but in order to not have automated tools unhappy, would it be possible to make a new release, given current master already depends on libtest-mimic 0.4, which depends on crossbeam-channel 0.5, thus fixing the issue?

Anyway, thank you for bolero once again!

EDIT: BTW, I also noticed there's no tag for cargo-bolero-0.7.0, which makes it harder for packagers to update the distro packages that provide it. I'm maintaining the nixos one, would it be possible for you to take advantage of this and add a tag on the commit on which cargo-bolero-0.7.0 was released? :)

Support RUSTC_BOOTSTRAP

This will enable support for sanitizers with stable builds (see #71)

Where should new implementations be placed?

I want to implement TypeGenerator on PathBuf and make a merge request, but I can't tell where it should be in Bolero's code. I know it should be in bolero-generator/src somewhere, but while it looks like you have std in there, most of the interesting stuff seems to be happening in the alloc directory. How do you plan on organizing this?

gen-arbitrary is slow

Hey! So I recently tried deriving TypeGenerator with a reasonably small type.

Basically, this was like using bolero::check!().with_type::<Vec<(#[gen_arbitrary()] Uuid, String, String)>>().

Turns out just doing this led to the test taking more time than I would wait just to generate the first input (I bisected with a panic!() and ended up at the top of the check, because I was convinced I had a deadlock somewhere). perf confirms, almost all the time was being spent in generating Strings.

Sure this was using cargo test, so without optimizations on. But still, maybe it'd make sense to have the default TypeGenerator for basic types be somewhat limited in size, so they don't literally take minutes to generate with relatively simple types? (I must say I have not tried actually profiling when fuzzing yet, so I don't know how badly this affects fuzzers yet, but for tests not using gen_with basically made the generated code impossible-to-use)

Implement test attribute

This would allow for more concise stateless tests:

#[bolero::test]
fn my_test(value: u64) {
    // checks go here
}

#[bolero::test(generator = my_generator())]
fn my_test(value: u64) {
    // checks go here
}

camshaft / bolero Goto Github PK

bolero's People

Contributors

Stargazers

Watchers

Forkers

bolero's Issues

Meta

Context

Recommend Projects

Recommend Topics

Recommend Org