`check` doesn't return `Bool` when counterexamples found about propcheck.jl HOT 2 CLOSED

You... you've got another packing coming that does PropCheck.jl stuff but better? Well then, I'm excited for that!

As for what check itself returns, I'm fine with its user-facing API to stay as it is for now. As for my wrapper, here's an improved update for anyone looking for a workaround. I understand why it was designed that way, I just felt the need to point out the discord between your documentation of usage examples and idiomatic Julia testing.

function wrap_check(prop::Function, gen::Integrated)
    chk = check(splat(prop), gen)
    chk isa Bool && return chk
    @info chk
    return prop(chk[1]...)
end

It grabs the counterexample and feeds it back to the property-checking function. So I can have @test wrap_check(...) throughout my test code, and upon finding counterexamples it'll have run the counterexample on prop and returned the false value with a nicer report than a test error.

Keep up the great work @Seelengrab

from propcheck.jl.

Loving the package so far!

Thank you! Nice to hear that someone is using this :)

Since check only returns true or the value of the counter example, I don't think this is appropriate usage.

This is intentional - it was the easiest way of getting the found counterexample reported as part of a testsuite, making use of @testset. As is, check should only ever return a Bool if the result is true, and a non-Bool counterexample otherwise. This is subtly broken if the counterexample itself is true:

julia> using PropCheck

julia> prop(b) = !b
prop (generic function with 1 method)

julia> using Test

julia> PropCheck.check(prop, PropCheck.itype(Bool))
┌ Info: Found counterexample for 'prop', beginning shrinking...
└   Counterexample = true
[ Info: 1 counterexamples found for prop
true

julia> @test PropCheck.check(prop, PropCheck.itype(Bool))
┌ Info: Found counterexample for 'prop', beginning shrinking...
└   Counterexample = true
[ Info: 1 counterexamples found for prop
Test Passed

which is of course a bit bad. I intended to eventually replace this @test check approach entirely with a solution based on a custom AbstractTestSet, which would avoid that problem, but never got to it. Part of the reason is that support for printing of custom testsets was only recently merged into Julia, see JuliaLang/julia#53215. Another reason is that I'm currently working on a package that already handles this & much more better, while being faster :)

As for changing what check itself returns - that likely won't change at all, if I'm going to change the user-facing API it'd be by replacing @ŧest with a different macro entirely.

from propcheck.jl.