Testing about plots.jl HOT 3 CLOSED

In general Pkg.test should run a lot of tests in the first place to test functionality and have good coverage. In this case everything of Plots.jl should be covered/called, not necessarily the backends. Some work could be spend to generate the tests out of combinations of options.

For regression testing (i.e. taking care you get the same output) i'd say take two backends with different rendering technique which are resonably stable (Gadfly as Cairo based, UnicodePlots as different) and just do plain image (png) comparison.

from plots.jl.

Yeah I think I agree that there should be two different sets of tests. The runtests.jl file should contain lots of Plots-only tests that test how input data is handled, that the data is sliced up correctly, that arguments flow through as expected, etc. No plots should be generated (or at most, they could be generated just to see if errors are thrown).

The second set of tests will compare actual output images, and should be started manually by someone who will examine differences between flagged images and handle appropriately (either confirm there's a regression/error, or decide that the new image is correct and replace the old reference image).

from plots.jl.

The testing framework is in place. I use @timholy 's image compare macro, and have methods to run through the examples and compare to reference images (for example, saved in test/refimg/v0.4/gadfly/ref1.png). If there is a difference, then a new Gtk window is built, showing the new image vs the reference image, and giving the options to skip or to save this image as the new reference.

In non-interactive mode, the error is simply rethrown. I use FactCheck, and have incorporated each example as a @fact, so that Travis will be able to perform full integration testing. (in theory!!)

There are some bugs to iron out, but does this framework sound like something that should end up in it's own package?

code: https://github.com/tbreloff/Plots.jl/blob/dev/test/imgcomp.jl

from plots.jl.