seelengrab / propcheck.jl Goto Github PK
View Code? Open in Web Editor NEWA package for simple property based testing in julia.
Home Page: https://seelengrab.github.io/PropCheck.jl/
License: MIT License
A package for simple property based testing in julia.
Home Page: https://seelengrab.github.io/PropCheck.jl/
License: MIT License
filter(f, i, trim::Bool)
has one of these boolean flag arguments. I don't like these because they're hard to read and dirty up the functional design.
If you feel similarly, some ideas:
filter(f, i ; trim::Bool)
(easier to read, at least)filter(f, i)
and trimfilter(f, i)
(I tend to like this from a functional multiple dispatch design perspective)filter(f, i, TRIM)
(an enum)PropCheck.jl currently treats all errors ocurring during shrinking the same, as just another failure of the predicate, as if it had returned false
. It does record that an error happened, but it treats all these errors the same - in reality, it would be interesting to see whether shrinking could be made error-aware, so that once an error is hit, only executions also throwing that same error (and not just any failure of the predicate!) are considered "valid" shrinks.
At the moment, PropCheck.jl assumes the generators don't throw errors. This is suboptimal, due to requiring more stringent checks from users than may be necessary. It would be cool if PropCheck.jl could either try/catch errors during such generation, or provide a way to ensure throwing values are never triggered during generation.
These objects are printed with way too much detail; it needs to be trimmed down a bit. The method to implement here is the three-arg Base.show(::IO, ::MIME"text/plain", obj)
, with obj
being all subtypes of AbstractIntegrated
.
The question is - how much detail is sufficient? I'd at least like to prevent the humongous type parameters from being printed, simply due to the vast majority of them not being all that interesting. The ones that are interesting are return types.
Right now, itype
doesn't specify what to do when a non-concrete type is requested - it'll likely error. It would be nice to be able to give itype
an abstract type, and have PropCheck.jl automatically generate valid instances for that type through reflection, providing a (very generic) integrated shrinker. This would also allow practical generation of Any
value.
The idea is that, given a function and a signature, it ought to be possible to automatically fuzz the method in question, by creating itype
generators for each argument and interleave
ing them.
There is also an adjacent, possibly required, objective here - a default generation of instances of a type, purely from that type. This would be done by checking whether there are any methods
attached to the given type, taking one random one & recursively generating the input arguments to that method. This will be horrible in the general case (constructors are for rejecting invalid arguments after all, so those shouldn't be put into constructors in the first place), but could be a nice idea for some code.
Once a failure is found, it'd be good to record the failure and replay it later on, to catch regressions of known-faulty inputs.
Some considerations about this:
Loving the package so far!
In the documented examples, expressions of the form
@test check(...)
are provided.
Since check
only returns true
or the value of the counter example, I don't think this is appropriate usage. I've had to write my own wrapper for my use cases. The one I found that works for me is
function wrap_check(prop::Function, gen::Integrated)
chk = check(prop, gen)
chk isa Bool && return chk
@test prop(chk)
return false
end
and then called it as
@test wrap_check(prop, gen)
I'm not too familiar with PropCheck.jl
internals. I think the type instability of check
's return values are somewhat justified for what it's trying to achieve. But I thought I might bring this up because people wanting to use your package would go down your package documentation's idiomatic usage and then travel down the road of discovery I went on.
On the other hand, we choose to be fine with PropCheck.jl
reporting counterexample finds as Expression evaluated to non-Boolean
test errors instead of test failures, the former of which I'm not a fan of.
The Julia style guide says
functions are lowercase (maximum, convert) and, when readable, with multiple words squashed together (isequal, haskey). When necessary, use underscores as word separators. Underscores are also used to indicate a combination of concepts (remotecall_fetch as a more efficient implementation of fetch(remotecall(...))) or as modifiers.
Would you mind applying that here? It's easier to learn and use an API if it follows the same conventions as the rest of the ecosystem and doesn't stick out in my code.
That would apply to shrinkTowards
and any others.
Just for your interest, I ran across these.
https://www.youtube.com/watch?v=uk6-ZHRu-5k
https://github.com/simonpoulding/DataGenerators.jl
https://github.com/simonpoulding/DataGeneratorTranslators.jl
I saw a conversation on Discourse about a logo for a PropCheck package badge. I'd been meaning to try my hand at some Luxor.jl, and thought this might be a good way to learn. No sweat if it isn't want you're looking for, just wanted to share in case it is helpful to you:
using Luxor
avg(x,y) = (x+y)/2
function parabolic_ish(start::Point, vertex::Point, stop::Point)
curve(
Point(start.x, avg(start.y, vertex.y)),
Point(avg(start.x, vertex.x), vertex.y),
vertex
)
curve(
Point(avg(vertex.x, stop.x), vertex.y),
Point(stop.x, avg(stop.y, vertex.y)),
stop
)
end
@draw begin
background("black")
inner_r = 50
space = 40
total_r = inner_r + space
blade_base_w = 40
half_w = blade_base_w / 2
blade_x_start = sqrt((total_r)^2 - (half_w)^2)
blade_len = 380
max_drag = 100
inner_curve_w = ((blade_len - blade_x_start) / 2.3) + blade_x_start
inner_curve_midpoint = avg(inner_curve_w, blade_x_start) * 1.1
blur_curve_mid_x = avg(inner_curve_w, blade_len)
colors = ("purple", "red", "green")
rotate(π/6)
for i in 1:3
sethue(colors[i])
move(Point(blade_x_start, -half_w))
# outer curve of the blade
curve(
Point(blade_len * 1.2, -(half_w + 200)),
Point(blade_len + 90, max_drag - 100),
Point(blade_len, max_drag))
# "blur" curve outer
parabolic_ish(
Point(blade_len, max_drag),
Point(avg(blade_len, blur_curve_mid_x), 20),
Point(blur_curve_mid_x, max_drag*.7)
)
# "blur" curve inner
parabolic_ish(
Point(blur_curve_mid_x, max_drag*.7),
Point(avg(blur_curve_mid_x, inner_curve_w), 20),
Point(inner_curve_w, max_drag*.4)
)
# inner most curve on bottom of blade
parabolic_ish(
Point(inner_curve_w, max_drag*.4),
Point(avg(inner_curve_w, blade_x_start), 20),
Point(blade_x_start, half_w)
)
arc(Point(0,0), total_r, sin(half_w/total_r), sin(half_w/total_r))
fillpath()
rotate(2π/3)
closepath()
end
rotate((π/3))
strokepath()
sethue("white")
move(Point(0,0))
sector(inner_r, total_r, 0, 2*π, :fill)
# circle(Point(0,0), inner_r, :fill)
end 1000 1000
An integrated shrinker like itype
currently produces all possible values of the given type, irrespective of the function the value ultimately ends up in. It would be interesting to see whether we can guide generation of values based on the branches on the generated value, in the function that is ultimately tested.
For example, in functions like this:
function foo(x::Int)
x < 5 && throw(ArgumentError("Not at least 5!"))
# other code
end
the input space is clearly delineated into two regions, those Int
smaller than 5 and thsoe at least 5. In the general case, these sorts of branch points are especially interesting, in particular when it comes to testing whether the branch point is correct. It would be great if PropCheck.jl provided not only integrated shrinkers, but also integrated generators - or function aware generators, which are biased to generate values close to these branch points.
This is a very out-there idea though, so this issue is just here for tracking the thought, not as an indicator that this is actively being worked on.
Some types have known special cases, e.g. Inf
for Float64
or ""
for String
. It'd be interesting to have an integrated shrinker that first generates these special cases, and only afterwards falls back to generating random instances of the desired type.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.