Option to shuffle order of it/describes about specta HOT 8 CLOSED

Fuck yes. I would like this very much.

from specta.

👍

from specta.

XCTest has full control over the order in which test case subclasses (i.e.: Specta specs) are executed. However, Specta should be able to randomize the order of the individual examples defined by those subclasses by shuffling the invocations returned by +[SPTXCTestCase testInvocations].

This shuffling should help catch examples that depend on previous examples in the same spec class. But it won't help to find examples that depend on examples in other spec classes.

It might be possible to shuffle the spec classes themselves by shuffling the classes returned by the swizzled +[XCTestCase xct_allSubclasses] method--which I believe was made public as of Xcode 6.

One question, @orta: you can randomize examples in RSpec using command line arguments: rspec --order random. How would you want to randomize examples in Specta? Command line arguments as well?

from specta.

Would it make sense to have the random generator seed be an (optional) argument as well? That way, you could replay a specific permutation to verify you fixed your inter-test dependencies.

from specta.

I think think a test launch argument probably makes the most sense

I'm happy for that to also shuffle the classes, the more shuffling the merrier IMO 👯

from specta.

will definitely add this to the todo list for 0.3

from specta.

from specta.

implemented in bcba04d

from specta.