awslabs / shuttle Goto Github PK

Shuttle is a library for testing concurrent Rust code

License: Apache License 2.0

Rust 100.00%

shuttle's Introduction

Shuttle

Shuttle is a library for testing concurrent Rust code. It is an implementation of a number of randomized concurrency testing techniques, including A Randomized Scheduler with Probabilistic Guarantees of Finding Bugs.

Getting started

Consider this simple piece of concurrent code:

use std::sync::{Arc, Mutex};
use std::thread;

let lock = Arc::new(Mutex::new(0u64));
let lock2 = lock.clone();

thread::spawn(move || {
    *lock.lock().unwrap() = 1;
});

assert_eq!(0, *lock2.lock().unwrap());

There is an obvious race condition here: if the spawned thread runs before the assertion, the assertion will fail. But writing a unit test that finds this execution is tricky. We could run the test many times and try to "get lucky" by finding a failing execution, but that's not a very reliable testing approach. Even if the test does fail, it will be difficult to debug: we won't be able to easily catch the failure in a debugger, and every time we make a change, we will need to run the test many times to decide whether we fixed the issue.

Randomly testing concurrent code with Shuttle

Shuttle avoids this issue by controlling the scheduling of each thread in the program, and scheduling those threads randomly. By controlling the scheduling, Shuttle allows us to reproduce failing tests deterministically. By using random scheduling, with appropriate heuristics, Shuttle can still catch most (non-adversarial) concurrency bugs even though it is not an exhaustive checker.

A Shuttle version of the above test just wraps the test body in a call to Shuttle's check_random function, and replaces the concurrency-related imports from std with imports from shuttle:

use shuttle::sync::{Arc, Mutex};
use shuttle::thread;

shuttle::check_random(|| {
    let lock = Arc::new(Mutex::new(0u64));
    let lock2 = lock.clone();

    thread::spawn(move || {
        *lock.lock().unwrap() = 1;
    });

    assert_eq!(0, *lock2.lock().unwrap());
}, 100);

This test detects the assertion failure with extremely high probability (over 99.9999%).

Shuttle is inspired by the Loom library for testing concurrent Rust code. Shuttle focuses on randomized testing, rather than the exhaustive testing that Loom offers. This is a soundness—scalability trade-off: Shuttle is not sound (a passing Shuttle test does not prove the code is correct), but it scales to much larger test cases than Loom. Empirically, randomized testing is successful at finding most concurrency bugs, which tend not to be adversarial.

License

This project is licensed under the Apache-2.0 License.

Security

See CONTRIBUTING for more information.

shuttle's People

Contributors

Stargazers

Watchers

shuttle's Issues

[TRACING] Improved testing of tracing functionality

See #128, in particular: #128 (comment).

Resolving this will involve:

Creating a custom tracing Subscriber which checks the traces emitted by the tests in tracing.rs (and optionally elsewhere).
Un-#[ignore]-ing the tests in tracing.rs.
Make the tests in tracing.rs always enable some LevelFilter (as in: overwriting the level set by RUST_LOG). Bonus points for making the tests try all different LevelFilters.
[OPTIONAL] Add more tests.

Suppress `generator` error on panic

When a thread panics in a Shuttle test, generator produces an error that gets printed to stdout via tracing. This error is a red herring: it mentions generator::gen_impl and so suggests that maybe something went wrong in that code, when really it's just that a Shuttle thread panicked and that panic is being propagated. It would be nice to suppress this error so we don't lead users down the wrong debugging path.

Failing schedule missing in output

I've seen the following failure output in CI runs. Without the failing schedule I'm unable to reproduce the failure.

thread 'my_shuttle_test' panicked at 'exceeded max_steps bound 1000000. this might be caused by an unfair schedule (e.g., a spin loop)?
pass that string to `shuttle::replay` to replay the failure', /path/to/shuttle/src/runtime/execution.rs:143:17
stack backtrace:
   ...

I'm quite baffled, since the error message must come from

shuttle/src/runtime/failure.rs

Lines 74 to 77 in 0310685

    
           format!( 
        
               "{}\nfailing schedule: \"{}\"\npass that string to `shuttle::replay` to replay the failure", 
        
               message, serialized_schedule 
        
           )

and I don't understand how the failing schedule part can be lost.

Early stopping doesn't work if a Drop impl tries to acquire locks

This test panics in the Drop impl of PoolItem:

#[test]
fn max_steps_continue_lock_during_drop() {
    let mut config = Config::new();
    config.max_steps = MaxSteps::ContinueAfter(10);
    let scheduler = DfsScheduler::new(None, false);
    let runner = Runner::new(scheduler, config);
    runner.run(|| {
        #[derive(Clone)]
        struct Pool {
            items: Arc<Mutex<VecDeque<usize>>>,
        }

        struct PoolItem {
            pool: Arc<Mutex<VecDeque<usize>>>,
            item: usize,
        }

        impl Pool {
            fn new(length: usize) -> Self {
                Self {
                    items: Arc::new(Mutex::new((0..length).collect())),
                }
            }

            fn get(&self) -> Option<PoolItem> {
                let mut items = self.items.lock().unwrap();
                let item = items.pop_front()?;
                Some(PoolItem {
                    pool: self.items.clone(),
                    item,
                })
            }
        }

        impl Drop for PoolItem {
            fn drop(&mut self) {
                let mut items = self.pool.lock().unwrap();
                items.push_back(self.item);
            }
        }

        let pool = Pool::new(10);

        let threads: Vec<_> = (0..3).map(|_| {
            let pool = pool.clone();
            thread::spawn(move || {
                let _item = pool.get();
                thread::yield_now();
            })
        }).collect();

        for thd in threads {
            thd.join().unwrap();
        }
    })
}

The problem is that ContinueAfter makes us stop the test early, while some PoolItems are still alive. During cleanup we drop the Generator for each thread, which tries to safely clean up by unwinding their stacks, including dropping the PoolItem. But PoolItem's drop tries to acquire a lock, which is no longer allowed once ContinueAfter has triggered.

I'm not totally sure what to do here:

We could just leak the continuation's stack...
We could do something smarter during teardown, allowing threads to continue running until termination or deadlock, under the assumption that the only code that can run once we drop the continuations is cleanup code

Is shuttle planning to support fork mode?

My application mutates global states (e.g., allocators, buffer pools) and may benefit from testing with separate processes. I'm wondering if it's possible for the shuttle to support fork mode. The first step could be making the PortfolioRunner run in different processes.

Provide more useful feedback for deadlocks

From @bkragl in #66:

One thing I was thinking about is whether it makes sense to have a feature in Shuttle that does precise on-the-fly deadlock detection. Right now there might be a deadlock among some tasks early on in an execution, but we need to wait until the end of the execution to detect it. It might be helpful for debugging to stop the execution right when the deadlock happens.

The nice thing about the current check is that it is completely generic. What I'm proposing would require knowledge about the synchronization primitives (e.g., a combined resource allocation graph for both Mutex and RwLock).

An intermediate step would be to print stack traces for deadlocked threads, so at least you know where to start the debugging process (e.g., which locks are involved).

Rayon support

Hi! How to use shuttle to test packages based on Rayon? My understanding is that shuttle has no direct support for Rayon primitives, so we need to change source code of Rayon. Is it true? Or is there any existing code base of Rayon adapted for shuttle?

Also, it will be great to add direct Rayon support.

Can't load replay schedule for max_steps failures

I have a shuttle test for a crate I wrote. It occasionally hits a deadlock that shuttle reports via hitting "exceeded max_steps bound ". It gives a (very big) failing schedule that I should pass to replay in order to reproduce the issue.

The problems are two fold:

replay_from_file can't load the outputted schedule string, always panicking with "invalid schedule"
reducing the max_steps via a custom Config.max_steps so that the schedule is able to be embedded as an argument to replay directly ends with shuttle erroring out with "expected context switch but next schedule step is random choice".

This unfortunately makes shuttle kind of useless for trying to fix this bug, since I can't exercise the reported deadlock to try and debug it under gdb or something to get a stacktrace of the stuck thread.

`wait_timeout` and friends shouldn't count as deadlocking

Shuttle complains that this test deadlocks, but in reality it doesn't:

#[test]
fn wait_timeout_deadlock() {
    check_dfs(
        || {
            let lock = Arc::new(Mutex::new(false));
            let cond = Arc::new(Condvar::new());

            let guard = lock.lock().unwrap();
            let (_guard, result) = cond.wait_timeout(guard, Duration::from_secs(1)).unwrap();
            assert!(result.timed_out());
        },
        None,
    )
}

The problem is that, while wait_timeout temporarily blocks the thread, it's not permanent, and so shouldn't count as deadlock.

We already knew that our modeling of wait_timeout wasn't complete because it doesn't test the timeout case, but this test shows the effects of such incompleteness—spurious failures.

We need to have a better notion of "blocked but can be unblocked". One cheap-ish idea would be for wait_timeout to spawn another (internal to Shuttle) "thread" that, when executed, causes the thread blocked in wait_timeout to unblock and return timeout. This would let the scheduler "naturally" decide when to trigger a timeout rather than us having to build any fancy time handling into the scheduler itself.

Replace Condvar epochs with clocks

#38 adds support for tracking causality using vector clocks. We think it might be possible for these vector clocks to replace epochs in the implementation of Condvar, since they capture the same information (an ordering between operations in the same condvar). This would hopefully make the Condvar implementation simpler and more obviously correct.

Release a new version?

Hi, first of all, thanks for the great project!

I try to use shuttle in one of my projects and it failed to resolve dependencies. The reason is that the version on crates.io pins its dependencies on minor versions. I noticed that it was fixed in this commit, but the code is not released to crates.io.

Is it possible to release a new minor version to reflect the changes made to shuttle since last release?

Make `future::JoinHandle::abort` actually cancel task

The implementation of abort in #87 as detaching the task does not seem right, because detached tasks may still continue to run. Consider the following test, which should not fail but currently does.

#[test]
fn join_handle_abort_bug() {
    check_dfs(
        || {
            let (sender, receiver) = futures::channel::oneshot::channel();
            let t = future::spawn({
                async move {
                    receiver.await.unwrap();
                    panic!("should not get here");
                }
            });
            t.abort();
            sender.send(()).unwrap();
            shuttle::thread::yield_now();
        },
        None,
    );
}

(The yield_now is needed because otherwise the main task would immediately finish, in which case also the execution finishes because there are no attached tasks left.)

We need a way to actually cancel a task, which drops its continuation and returns a JoinError indicating cancellation.

Implement thread park(), unpark(), and panicking()

Already a TODO in code:

shuttle/src/thread.rs

Line 173 in 4a00174

// TODO: Implement park(), unpark()

This would help making shuttle::thread to be an in-place substitute to std::thread.

[TRACING] Add test which would catch the linked panic

See #128 (comment).

Resolving this issue would involve creating a test which panics when run under #9a29bca, but does not panic when run under #c4c6517.

The issue seems to have been:

Task X enters Span A. Context-switch to Task Y.
Task Y enters Span A. Task Y leaves Span A, which closes it. Context switch to Task X.
Task X tries to enter Span A, which does not exist anymore.

May be more details needed, have not looked into this issue much.

`Once` breaks if moved

Here's a test that illustrates the bug:

#[test]
fn once_test() {
    crate::check_dfs(
        || {
            let once = Once::new();
            once.call_once(|| {});
            assert_eq!(is_completed_borrowed(&once), true);
            assert_eq!(is_completed_owned(once), true); // FAILS
        },
        None,
    );
}

fn is_completed_borrowed(once: &Once) -> bool {
    once.is_completed()
}

fn is_completed_owned(once: Once) -> bool {
    once.is_completed()
}

The issue is that we use the address of the Once to compute a storage key, which changes when the Once is moved.

shuttle/src/sync/once.rs

Lines 166 to 170 in 2ac1fae

    
           impl From<&Once> for StorageKey { 
        
               fn from(once: &Once) -> Self { 
        
                   StorageKey(once as *const _ as usize, 0x2) 
        
               } 
        
           }

Fix tracing spans

#99 did a partial fix of the tracing issue. Traces are now a lot better than they used to be, but are still not entirely correct. For instance, the following code:

use shuttle::sync::{Arc, Mutex};
use tracing::{warn_span, warn};
use test_log::test;
use shuttle::{check_random, thread};

#[test]
fn tracing_nested_spans() {
    check_random(|| {
        let lock = Arc::new(Mutex::new(0));
        let threads: Vec<_> = (0..3).map(|i| {
            let lock = lock.clone();
            thread::spawn(move || {
                let outer = warn_span!("outer", tid=i);
                let _outer = outer.enter();
                {
                    let mut locked = lock.lock().unwrap();
                    let inner = warn_span!("inner", tid=i);
                    let _inner = inner.enter();
                    warn!("incrementing from {}", *locked);
                    *locked += 1;
                }
            })
        }).collect();

        for thread in threads {
            thread.join().unwrap();
        }
    }, 1)
}

Gives the following output:

running 1 test
2023-04-20T21:21:17.499960Z TRACE execution{i=0}: shuttle::runtime::execution: runnable=[TaskId(0)] next_task=Some(TaskId(0))
2023-04-20T21:21:17.501096Z TRACE execution{i=0}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(1)] next_task=Some(TaskId(1))
2023-04-20T21:21:17.501338Z TRACE execution{i=0}:step{i=1 task=1}:outer{tid=0}: shuttle::sync::mutex: waiting to acquire mutex 0x150e0bce0 holder=None waiters=TaskSet { }
2023-04-20T21:21:17.501403Z TRACE execution{i=0}:step{i=1 task=1}:outer{tid=0}: shuttle::sync::mutex: acquired mutex 0x150e0bce0 waiters=TaskSet { }
2023-04-20T21:21:17.501667Z TRACE execution{i=0}:step{i=1 task=1}:outer{tid=0}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(1)] next_task=Some(TaskId(0))
2023-04-20T21:21:17.501747Z TRACE execution{i=0}:step{i=1 task=1}:outer{tid=0}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(1), TaskId(2)] next_task=Some(TaskId(1))
2023-04-20T21:21:17.501843Z  WARN execution{i=0}:step{i=1 task=1}:inner{tid=0}: r#mod::basic::uncontrolled_nondeterminism: incrementing from 0
2023-04-20T21:21:17.501984Z TRACE execution{i=0}:step{i=1 task=1}: shuttle::sync::mutex: releasing mutex 0x150e0bce0 waiters=TaskSet { }
2023-04-20T21:21:17.502159Z TRACE execution{i=0}:step{i=1 task=1}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(1), TaskId(2)] next_task=Some(TaskId(0))
2023-04-20T21:21:17.502233Z TRACE execution{i=0}:step{i=1 task=1}:outer{tid=0}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(1), TaskId(2), TaskId(3)] next_task=Some(TaskId(1))
2023-04-20T21:21:17.502355Z TRACE execution{i=0}:step{i=1 task=1}: shuttle::thread: thread finished, dropping thread locals
2023-04-20T21:21:17.502484Z TRACE execution{i=0}:step{i=1 task=1}: shuttle::thread: done dropping thread locals
2023-04-20T21:21:17.502769Z TRACE execution{i=0}:step{i=1 task=1}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(2), TaskId(3)] next_task=Some(TaskId(2))
2023-04-20T21:21:17.502817Z TRACE execution{i=0}:step{i=3 task=2}:outer{tid=1}: shuttle::sync::mutex: waiting to acquire mutex 0x150e0bce0 holder=None waiters=TaskSet { }
2023-04-20T21:21:17.502849Z TRACE execution{i=0}:step{i=3 task=2}:outer{tid=1}: shuttle::sync::mutex: acquired mutex 0x150e0bce0 waiters=TaskSet { }
2023-04-20T21:21:17.502887Z TRACE execution{i=0}:step{i=3 task=2}:outer{tid=1}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(2), TaskId(3)] next_task=Some(TaskId(3))
2023-04-20T21:21:17.502936Z TRACE execution{i=0}:step{i=5 task=3}:outer{tid=2}: shuttle::sync::mutex: waiting to acquire mutex 0x150e0bce0 holder=Some(TaskId(2)) waiters=TaskSet { }
2023-04-20T21:21:17.503072Z TRACE execution{i=0}:step{i=5 task=3}:outer{tid=2}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(2)] next_task=Some(TaskId(0))
2023-04-20T21:21:17.503271Z TRACE execution{i=0}:step{i=5 task=3}:outer{tid=2}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(2)] next_task=Some(TaskId(2))
2023-04-20T21:21:17.503317Z  WARN execution{i=0}:step{i=3 task=2}:inner{tid=1}: r#mod::basic::uncontrolled_nondeterminism: incrementing from 1
2023-04-20T21:21:17.503348Z TRACE execution{i=0}:step{i=3 task=2}: shuttle::sync::mutex: releasing mutex 0x150e0bce0 waiters=TaskSet { 3 }
2023-04-20T21:21:17.503447Z TRACE execution{i=0}:step{i=3 task=2}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(2), TaskId(3)] next_task=Some(TaskId(2))
2023-04-20T21:21:17.503484Z TRACE execution{i=0}:step{i=3 task=2}: shuttle::thread: thread finished, dropping thread locals
2023-04-20T21:21:17.503504Z TRACE execution{i=0}:step{i=3 task=2}: shuttle::thread: done dropping thread locals
2023-04-20T21:21:17.503528Z TRACE execution{i=0}:step{i=3 task=2}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(3)] next_task=Some(TaskId(3))
2023-04-20T21:21:17.503557Z TRACE execution{i=0}:step{i=5 task=3}: shuttle::sync::mutex: acquired mutex 0x150e0bce0 waiters=TaskSet { }
2023-04-20T21:21:17.503590Z TRACE execution{i=0}:step{i=5 task=3}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(3)] next_task=Some(TaskId(3))
2023-04-20T21:21:17.503626Z  WARN execution{i=0}:step{i=5 task=3}:inner{tid=2}: r#mod::basic::uncontrolled_nondeterminism: incrementing from 2
2023-04-20T21:21:17.503657Z TRACE execution{i=0}:step{i=5 task=3}: shuttle::sync::mutex: releasing mutex 0x150e0bce0 waiters=TaskSet { }
2023-04-20T21:21:17.503688Z TRACE execution{i=0}:step{i=5 task=3}: shuttle::runtime::execution: runnable=[TaskId(0), TaskId(3)] next_task=Some(TaskId(3))
2023-04-20T21:21:17.503719Z TRACE execution{i=0}:step{i=5 task=3}: shuttle::thread: thread finished, dropping thread locals
2023-04-20T21:21:17.503738Z TRACE execution{i=0}:step{i=5 task=3}: shuttle::thread: done dropping thread locals
2023-04-20T21:21:17.503759Z TRACE execution{i=0}:step{i=5 task=3}: shuttle::runtime::execution: runnable=[TaskId(0)] next_task=Some(TaskId(0))
2023-04-20T21:21:17.503905Z TRACE execution{i=0}: shuttle::runtime::execution: runnable=[TaskId(0)] next_task=Some(TaskId(0))
2023-04-20T21:21:17.503934Z TRACE execution{i=0}: shuttle::runtime::execution: runnable=[TaskId(0)] next_task=Some(TaskId(0))
2023-04-20T21:21:17.504147Z TRACE execution{i=0}: shuttle::thread: thread finished, dropping thread locals
2023-04-20T21:21:17.504245Z TRACE execution{i=0}: shuttle::thread: done dropping thread locals
2023-04-20T21:21:17.504908Z  INFO shuttle::scheduler::metrics: run finished iterations=1 steps=[min=17, max=17, avg=17.0] context_switches=[min=11, max=11, avg=11.0] preemptions=[min=7, max=7, avg=7.0] random_choices=[min=0, max=0, avg=0.0]
test basic::tracing::tracing_nested_spans ... ok

when run under Shuttle

If we run it outside of Shuttle, by changing it to the following:

#[test]
fn tracing_nested_spans() {
        let lock = Arc::new(std::sync::Mutex::new(0));
        let threads: Vec<_> = (0..3).map(|i| {
            let lock = lock.clone();
            std::thread::spawn(move || {
                let outer = warn_span!("outer", tid=i);
                let _outer = outer.enter();
                {
                    let mut locked = lock.lock().unwrap();
                    let inner = warn_span!("inner", tid=i);
                    let _inner = inner.enter();
                    warn!("incrementing from {}", *locked);
                    *locked += 1;
                }
            })
        }).collect();

        for thread in threads {
            thread.join().unwrap();
        }
}

then we get:

running 1 test
2023-04-20T21:26:45.242960Z  WARN outer{tid=0}:inner{tid=0}: r#mod::basic::uncontrolled_nondeterminism: incrementing from 0
2023-04-20T21:26:45.243031Z  WARN outer{tid=1}:inner{tid=1}: r#mod::basic::uncontrolled_nondeterminism: incrementing from 1
2023-04-20T21:26:45.243085Z  WARN outer{tid=2}:inner{tid=2}: r#mod::basic::uncontrolled_nondeterminism: incrementing from 2
test basic::tracing::tracing_nested_spans ... ok

In other words: the current scheme only shows the head of the current trace stack (just inner), while it should show the entirety of the stack (outer then inner). This can of course be reconstructed afterwards, but it would be nice if we emitted the correct trace by default.

shuttle::thread_local isn't dropped when threads exit the scheduler

I have a crate that setups up cross-thread channels via thread_local and lazy_static. Shuttle reports a spurious deadlock, however, because it only frees the thread_local value for threads when they actually exit and not when they "exit" the thread scheduler (such as hitting the end of shuttle::check_random(|| { })) - I am using the thread_local Drop impl causing the last sender of a channel to go away to unblock another thread's receiver, so the other thread can exit once there are no more clients. Because thread_locals don't drop, the sender is never removed, and so the receiver is reported as a deadlock, despite this not being possible in normal operations.

I have a manual workaround, where I just RefCell::take() the channel at the end of my shuttle tests in order to manually drop the sender, but it would be nice to not need this (and it took me more than a little while to figure out what exactly was going on and realize the problem, which may hit other people).

"panicked while panicking" issues when drop handlers run concurrent code

Shuttle does not properly handle concurrent code run in drop handlers after a panic. It can run into a panic while panicking, which kills the process. Here's a minimal example. It needs to hit a particular schedule which happens with probability 1/2.

use shuttle::sync::{Arc, Mutex};

struct Foo(Arc<Mutex<()>>);

impl Drop for Foo {
    fn drop(&mut self) {
        let _m = self.0.lock().unwrap();
    }
}

#[test]
fn double_panic() {
    shuttle::check_random(
        || {
            let mutex = Arc::new(Mutex::new(()));
            let _f = Foo(mutex.clone());

            shuttle::thread::spawn(move || {
                // The double panic happens in executions where this thread
                // switches to the main thread inside the `lock` function.
                let _m = mutex.lock().unwrap();
            });

            panic!("panic in test code");
        },
        100,
    );
}

Error output

test panicked in task 'main-thread'
failing schedule:
"
910103a4d0e880c2fbe3a89f0108
"
pass that string to `shuttle::replay` to replay the failure
thread 'double_panic' panicked at 'panic in test code', src/lib.rs:24:13
stack backtrace:
   0: rust_begin_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
   1: core::panicking::panic_fmt
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
   2: proptest_shuttle_test::double_panic::{{closure}}
             at ./src/lib.rs:24:13
   3: shuttle::runtime::runner::Runner<S>::run::{{closure}}::{{closure}}::{{closure}}
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/runner.rs:69:100
   4: shuttle::thread::thread_fn
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/thread.rs:102:15
   5: shuttle::runtime::execution::Execution::run::{{closure}}::{{closure}}
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:67:25
   6: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
   7: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
   8: shuttle::runtime::thread::continuation::Continuation::new::{{closure}}
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/thread/continuation.rs:87:21
   9: generator::gen_impl::GeneratorImpl<A,T>::init_code::{{closure}}
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:336:21
  10: generator::stack::StackBox<F>::call_once
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/stack/mod.rs:139:13
  11: generator::stack::Func::call_once
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/stack/mod.rs:121:9
  12: generator::gen_impl::gen_init::{{closure}}
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:552:9
  13: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
  14: std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  15: ___rust_try
  16: std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  17: std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  18: generator::gen_impl::catch_unwind_filter
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:543:5
  19: generator::gen_impl::gen_init
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:570:25
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'double_panic' panicked at 'test panicked in task 'main-thread'
failing schedule:
"
910103a4d0e880c2fbe3a89f0108
"
pass that string to `shuttle::replay` to replay the failure', /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:144:17
stack backtrace:
   0:        0x10f665546 - std::backtrace_rs::backtrace::libunwind::trace::h0a908cd09b5a35f9
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:        0x10f665546 - std::backtrace_rs::backtrace::trace_unsynchronized::hc0e7d5d16c14a788
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x10f665546 - std::sys_common::backtrace::_print_fmt::hcf9ca6805c7eb2fe
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:65:5
   3:        0x10f665546 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hae51cb91d407e2ef
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x10f68103b - core::fmt::write::h746bc0969202388b
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/fmt/mod.rs:1254:17
   5:        0x10f66250c - std::io::Write::write_fmt::h4098c2c7437a0bd7
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/io/mod.rs:1698:15
   6:        0x10f66531a - std::sys_common::backtrace::_print::he6d3aef1f6c73e2d
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:47:5
   7:        0x10f66531a - std::sys_common::backtrace::print::h8360bf0158e89b36
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:34:9
   8:        0x10f666f70 - std::panicking::default_hook::{{closure}}::hedf04c568eb6e0bc
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:269:22
   9:        0x10f666d20 - std::panicking::default_hook::h62889b2c29e2347d
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:288:9
  10:        0x10f6255c4 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::he6b83307c909e624
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1987:9
  11:        0x10f5fe425 - shuttle::runtime::failure::init_panic_hook::{{closure}}::{{closure}}::h42de85c8fa97934f
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/failure.rs:151:13
  12:        0x10f6255c4 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::he6b83307c909e624
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1987:9
  13:        0x10f6269e5 - generator::gen_impl::catch_unwind_filter::{{closure}}::{{closure}}::h24c0991694e38682
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:539:13
  14:        0x10f66764a - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h5ccd129639ee877c
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1987:9
  15:        0x10f66764a - std::panicking::rust_panic_with_hook::h22edd02828aff274
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:695:13
  16:        0x10f6673e4 - std::panicking::begin_panic_handler::{{closure}}::h6895f44a9b7e2caa
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:582:13
  17:        0x10f665989 - std::sys_common::backtrace::__rust_end_short_backtrace::h013519e9b77978c5
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:150:18
  18:        0x10f66712d - rust_begin_unwind
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
  19:        0x10f68d053 - core::panicking::panic_fmt::h2a51b60c12d11322
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
  20:        0x10f5a7221 - core::panicking::panic_display::hbd949b297acfd5f4
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:150:5
  21:        0x10f5a6ae1 - shuttle::runtime::execution::Execution::step::h00314f3fd4ea0f15
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:144:17
  22:        0x10f59d89f - shuttle::runtime::execution::Execution::run::{{closure}}::h435d08498b5500db
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:74:19
  23:        0x10f5a92b9 - scoped_tls::ScopedKey<T>::set::hf9c179dc9c5d2b9e
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/scoped-tls-1.0.1/src/lib.rs:137:9
  24:        0x10f59d6d1 - shuttle::runtime::execution::Execution::run::hb1f06acf42b88a14
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:64:9
  25:        0x10f593d2a - shuttle::runtime::runner::Runner<S>::run::{{closure}}::{{closure}}::h62c8cb787820efb7
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/runner.rs:69:64
  26:        0x10f592555 - tracing::span::Span::in_scope::h9482e775058da9fb
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/span.rs:1102:9
  27:        0x10f593bbe - shuttle::runtime::runner::Runner<S>::run::{{closure}}::h5b30f2386550abea
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/runner.rs:69:17
  28:        0x10f5a9197 - scoped_tls::ScopedKey<T>::set::h46c950750c8aadb1
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/scoped-tls-1.0.1/src/lib.rs:137:9
  29:        0x10f5932c3 - shuttle::runtime::runner::Runner<S>::run::h6ed27f200638e525
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/runner.rs:49:9
  30:        0x10f5a1159 - shuttle::check_random::hf2b9458e9d07ee53
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/lib.rs:313:5
  31:        0x10f59223e - proptest_shuttle_test::double_panic::h236e26cd0e9ef228
                               at /Users/kraglb/Code/proptest-shuttle-test/src/lib.rs:13:5
  32:        0x10f5a6fb9 - proptest_shuttle_test::double_panic::{{closure}}::h803d69b63cde3b8b
                               at /Users/kraglb/Code/proptest-shuttle-test/src/lib.rs:12:19
  33:        0x10f594658 - core::ops::function::FnOnce::call_once::hc924793af4a1d820
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
  34:        0x10f5e34c2 - core::ops::function::FnOnce::call_once::h0fe66ffa3f9dbea8
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
  35:        0x10f5e34c2 - test::__rust_begin_short_backtrace::h460edc73ee3d7292
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/test/src/lib.rs:655:18
  36:        0x10f5b0281 - test::run_test::{{closure}}::hcb97f2f14ca1b43f
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/test/src/lib.rs:646:30
  37:        0x10f5b0281 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hdfe6b8e984779cce
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
  38:        0x10f5e22c1 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hfa918f0892d6e1a1
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
  39:        0x10f5e22c1 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h71c8bbcc51698713
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panic/unwind_safe.rs:271:9
  40:        0x10f5e22c1 - std::panicking::try::do_call::h890697a79ad71324
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  41:        0x10f5e22c1 - std::panicking::try::ha3f7932050f4d091
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  42:        0x10f5e22c1 - std::panic::catch_unwind::hc2bb78c2421caad4
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  43:        0x10f5e22c1 - test::run_test_in_process::hfaa808ccf4afcea3
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/test/src/lib.rs:678:27
  44:        0x10f5e22c1 - test::run_test::run_test_inner::{{closure}}::h6df30ee50946e418
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/test/src/lib.rs:572:39
  45:        0x10f5aaba7 - test::run_test::run_test_inner::{{closure}}::h28731cdeaae649ee
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/test/src/lib.rs:599:37
  46:        0x10f5aaba7 - std::sys_common::backtrace::__rust_begin_short_backtrace::he77a8a07f5d086a1
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  47:        0x10f5b0151 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h7325a38e91958334
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/mod.rs:526:17
  48:        0x10f5b0151 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4f76313c8805442a
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panic/unwind_safe.rs:271:9
  49:        0x10f5b0151 - std::panicking::try::do_call::h8ffd919c7cd22e24
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  50:        0x10f5b0151 - std::panicking::try::he40bc179d7d75f43
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  51:        0x10f5b0151 - std::panic::catch_unwind::hb878b7a33e4fb646
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  52:        0x10f5b0151 - std::thread::Builder::spawn_unchecked_::{{closure}}::hf2774d6144cce2f7
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/mod.rs:525:30
  53:        0x10f5b0151 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc09679af1629946e
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
  54:        0x10f66b8a9 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h3813d4e7f4f8581d
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
  55:        0x10f66b8a9 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::ha3873a2c5aad6d58
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
  56:        0x10f66b8a9 - std::sys::unix::thread::Thread::new::thread_start::ha9b8d2206382b425
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys/unix/thread.rs:108:17
  57:     0x7ff8045c94e1 - __pthread_start
thread panicked while panicking. aborting.
error: test failed, to rerun pass `-p proptest-shuttle-test --lib`

Caused by:
  process didn't exit successfully: `/Users/kraglb/Code/proptest-shuttle-test/target/debug/deps/proptest_shuttle_test-cf7e6e3a9da694c9 double_panic --exact --nocapture` (signal: 6, SIGABRT: process abort signal)

Replaying the above schedule (910103a4d0e880c2fbe3a89f0108) also leads to a panic while panicking, because the schedule stops at the first panic, and so we hit 'schedule ended early' when we want to make further scheduling decisions after the first panic.

Replay error output

test panicked in task 'main-thread'
failing schedule:
"
910103a4d0e880c2fbe3a89f0108
"
pass that string to `shuttle::replay` to replay the failure
thread 'replay_double_panic' panicked at 'panic in test code', src/lib.rs:43:13
stack backtrace:
   0: rust_begin_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
   1: core::panicking::panic_fmt
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
   2: proptest_shuttle_test::replay_double_panic::{{closure}}
             at ./src/lib.rs:43:13
   3: shuttle::runtime::runner::Runner<S>::run::{{closure}}::{{closure}}::{{closure}}
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/runner.rs:69:100
   4: shuttle::thread::thread_fn
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/thread.rs:102:15
   5: shuttle::runtime::execution::Execution::run::{{closure}}::{{closure}}
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:67:25
   6: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
   7: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
   8: shuttle::runtime::thread::continuation::Continuation::new::{{closure}}
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/thread/continuation.rs:87:21
   9: generator::gen_impl::GeneratorImpl<A,T>::init_code::{{closure}}
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:336:21
  10: generator::stack::StackBox<F>::call_once
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/stack/mod.rs:139:13
  11: generator::stack::Func::call_once
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/stack/mod.rs:121:9
  12: generator::gen_impl::gen_init::{{closure}}
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:552:9
  13: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
  14: std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  15: ___rust_try
  16: std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  17: std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  18: generator::gen_impl::catch_unwind_filter
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:543:5
  19: generator::gen_impl::gen_init
             at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:570:25
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'replay_double_panic' panicked at 'schedule ended early', /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/scheduler/replay.rs:69:13
stack backtrace:
   0:        0x10f3e3036 - std::backtrace_rs::backtrace::libunwind::trace::h0a908cd09b5a35f9
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:        0x10f3e3036 - std::backtrace_rs::backtrace::trace_unsynchronized::hc0e7d5d16c14a788
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x10f3e3036 - std::sys_common::backtrace::_print_fmt::hcf9ca6805c7eb2fe
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:65:5
   3:        0x10f3e3036 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hae51cb91d407e2ef
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x10f3feb2b - core::fmt::write::h746bc0969202388b
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/fmt/mod.rs:1254:17
   5:        0x10f3dfffc - std::io::Write::write_fmt::h4098c2c7437a0bd7
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/io/mod.rs:1698:15
   6:        0x10f3e2e0a - std::sys_common::backtrace::_print::he6d3aef1f6c73e2d
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:47:5
   7:        0x10f3e2e0a - std::sys_common::backtrace::print::h8360bf0158e89b36
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:34:9
   8:        0x10f3e4a60 - std::panicking::default_hook::{{closure}}::hedf04c568eb6e0bc
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:269:22
   9:        0x10f3e4810 - std::panicking::default_hook::h62889b2c29e2347d
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:288:9
  10:        0x10f3a2cf4 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::he6b83307c909e624
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1987:9
  11:        0x10f377ed5 - shuttle::runtime::failure::init_panic_hook::{{closure}}::{{closure}}::h42de85c8fa97934f
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/failure.rs:151:13
  12:        0x10f3a2cf4 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::he6b83307c909e624
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1987:9
  13:        0x10f3a4115 - generator::gen_impl::catch_unwind_filter::{{closure}}::{{closure}}::h24c0991694e38682
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:539:13
  14:        0x10f3e513a - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h5ccd129639ee877c
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1987:9
  15:        0x10f3e513a - std::panicking::rust_panic_with_hook::h22edd02828aff274
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:695:13
  16:        0x10f3e4e93 - std::panicking::begin_panic_handler::{{closure}}::h6895f44a9b7e2caa
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:580:13
  17:        0x10f3e3479 - std::sys_common::backtrace::__rust_end_short_backtrace::h013519e9b77978c5
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:150:18
  18:        0x10f3e4c1d - rust_begin_unwind
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
  19:        0x10f40ab43 - core::panicking::panic_fmt::h2a51b60c12d11322
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
  20:        0x10f376656 - <shuttle::scheduler::replay::ReplayScheduler as shuttle::scheduler::Scheduler>::next_task::h8ea057550dca6f79
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/scheduler/replay.rs:69:13
  21:        0x10f304bd6 - <shuttle::scheduler::metrics::MetricsScheduler<S> as shuttle::scheduler::Scheduler>::next_task::hf2b74c3aa7bbc9ff
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/scheduler/metrics.rs:92:22
  22:        0x10f371630 - shuttle::runtime::execution::ExecutionState::schedule::h89fc2a09bfd49009
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:567:26
  23:        0x10f3708cd - shuttle::runtime::execution::ExecutionState::maybe_yield::{{closure}}::h98f52af81cfe20db
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:383:26
  24:        0x10f36fd8a - shuttle::runtime::execution::ExecutionState::try_with::{{closure}}::hb9dcd22ea989e68b
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:258:26
  25:        0x10f37e402 - scoped_tls::ScopedKey<T>::with::hdc6b8fddccec0df1
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/scoped-tls-1.0.1/src/lib.rs:171:13
  26:        0x10f36f366 - shuttle::runtime::execution::ExecutionState::try_with::h4bb2a7396c03ac15
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:256:13
  27:        0x10f36f06d - shuttle::runtime::execution::ExecutionState::with::ha3108570b68fa7dc
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:245:9
  28:        0x10f370899 - shuttle::runtime::execution::ExecutionState::maybe_yield::hdca648f3f72a96f5
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:377:9
  29:        0x10f363f9d - shuttle::runtime::thread::continuation::switch::h1ec241841f45c810
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/thread/continuation.rs:259:8
  30:        0x10f31d5ba - shuttle::sync::mutex::Mutex<T>::lock::hc5a83ab06bf8d63a
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/sync/mutex.rs:71:13
  31:        0x10f303a9d - <proptest_shuttle_test::Foo as core::ops::drop::Drop>::drop::hb411a29ef0c6f032
                               at /Users/kraglb/Code/proptest-shuttle-test/src/lib.rs:7:18
  32:        0x10f300f75 - core::ptr::drop_in_place<proptest_shuttle_test::Foo>::h04cb8183f3606aa5
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ptr/mod.rs:490:1
  33:        0x10f31c3ba - proptest_shuttle_test::replay_double_panic::{{closure}}::h91ac582f84cd9cfb
                               at /Users/kraglb/Code/proptest-shuttle-test/src/lib.rs:44:9
  34:        0x10f2ffa7d - shuttle::runtime::runner::Runner<S>::run::{{closure}}::{{closure}}::{{closure}}::h9e7bb64bc90b3ad9
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/runner.rs:69:100
  35:        0x10f3129d3 - shuttle::thread::thread_fn::hcbc2b34780f1e487
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/thread.rs:102:15
  36:        0x10f30df1a - shuttle::runtime::execution::Execution::run::{{closure}}::{{closure}}::h857f7da1b457d850
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/execution.rs:67:25
  37:        0x10f3002d4 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h1667835ca21b1394
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
  38:        0x10f36ea1e - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc62867ebb53a9c78
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
  39:        0x10f363431 - shuttle::runtime::thread::continuation::Continuation::new::{{closure}}::hd949b502d8cd8af8
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/shuttle-0.6.1/src/runtime/thread/continuation.rs:87:21
  40:        0x10f397039 - generator::gen_impl::GeneratorImpl<A,T>::init_code::{{closure}}::h71323f9c2b036413
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:336:21
  41:        0x10f38b33f - generator::stack::StackBox<F>::call_once::h856ea0bb4902f7b8
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/stack/mod.rs:139:13
  42:        0x10f3a2daf - generator::stack::Func::call_once::h7b0f46ad5849766c
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/stack/mod.rs:121:9
  43:        0x10f3a42c3 - generator::gen_impl::gen_init::{{closure}}::hb5197d56bc3a43dd
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:552:9
  44:        0x10f3a1115 - core::ops::function::FnOnce::call_once::h44414163bef47564
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
  45:        0x10f3a29a8 - std::panicking::try::do_call::hf38821f2c0f1e724
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  46:        0x10f3a2d1d - ___rust_try
  47:        0x10f3a2917 - std::panicking::try::h6b11b2e34ab64c77
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  48:        0x10f3a20b1 - std::panic::catch_unwind::ha56f94586618f63b
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  49:        0x10f3a3fe0 - generator::gen_impl::catch_unwind_filter::h345f378589a64220
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:543:5
  50:        0x10f3a414e - generator::gen_impl::gen_init::h25dfd0df328bbc96
                               at /Users/kraglb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/generator-0.7.4/src/gen_impl.rs:570:25
thread panicked while panicking. aborting.
error: test failed, to rerun pass `-p proptest-shuttle-test --lib`

Caused by:
  process didn't exit successfully: `/Users/kraglb/Code/proptest-shuttle-test/target/debug/deps/proptest_shuttle_test-cf7e6e3a9da694c9 replay_double_panic --exact --nocapture` (signal: 6, SIGABRT: process abort signal)

Use of Shuttle for Rust async and #[tokio::main]

Hi, I am very interested in what Shuttle brings to the table for testing.
As far as I can tell it is mostly designed for Rust sync code without #[tokio::main] main setups.

But I am a bit confused by the presences of the shuttle::future module.
I am guessing it's really for sync programs which are invoking a little bit of async code?

It seems like if Shuttle was intended for testing mostly async programs, there would
be replacement implementations for stuff like tokio::sync::Mutex (An asynchronous Mutex-like type).

Anyway I'd like to get some feedback about the suitability of Shuttle for testing mostly async functions and code.

Thank you. And thank you for creating Shuttle.

ReplayScheduler should have a new_from_file function

Right now we have shuttle::replay and shuttle::replay_from_file, which both call ReplayScheduler::new_from_encoded. That's inconvenient if you want to replay a schedule from a file but also pass in a shuttle::Config -- you can't pass a config into replay_from_file, so you have to parse the schedule yourself so that you can call ReplayScheduler::new_from_encoded with a schedule included.

We should either make replay/replay_from_file take a config as input or make a new_from_file function on ReplayScheduler (my preferred option).

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

	format!(
	"{}\nfailing schedule: \"{}\"\npass that string to `shuttle::replay` to replay the failure",
	message, serialized_schedule
	)

	impl From<&Once> for StorageKey {
	fn from(once: &Once) -> Self {
	StorageKey(once as *const _ as usize, 0x2)
	}
	}