Improving test harness: add "expected broken" feature about links HOT 12 CLOSED

Will we be merging branch https://github.com/links-lang/links/tree/js-expected-broken-tests ? This would give us a working Travis build. I admit it is not a fully satisfying solution, definitely one that does not scale easily to further extensions like conditional flags. But it will do as a temporary kludge.

from links.

I meant to ask about this branch on Tuesday but we ran out of time. I'd be happy with merging it.

from links.

Rebase probably won't be clean but that's I minor issue. I'll take care of it.

from links.

Here's a quick and dirty at fixing this ticket. For now I focused on database tests because we have one broken at the moment. The database tests are no longer picked automatically. Instead, they all need to be listed in a file tests/database/testsuite.config. Test name can be preceded with word BROKEN, which tells the run-tests script that this test is expected not work and it is OK for it to fail. Moreover, if a test marked as broken actually passes this will be reported as a failure.

Do others like this design from the perspective of someone who will be writing tests? Like I said, tests are no longer picked up automatically, they need to be explicitly "registered". Is that an acceptable price for this feature?

from links.

Registering tests seems okay to me.

Maybe we should discuss testing at the next meeting?

There are a number (likely growing) of switches that enable new features, like query shredding. There are tests that can only be run with some combination of features turned on (e.g. nested queries require shredding). Additionally, it would be great to rerun all the tests that didn't require (e.g.) shredding with shredding turned on anyways, to spot regressions.
With some features, some expected test results change, though. (E.g. relaxed constraints on what types are allowed in a query.)

There must be other projects with complicated testing. Can we perhaps steal their infrastructure?

from links.

Maybe we should discuss testing at the next meeting?

I won't be at the meeting on Friday - I have to go to the Jobcentre to apply for my National Insurance Number. Perhaps we could have this discussion during a lunch?

There must be other projects with complicated testing. Can we perhaps steal their infrastructure?

I already mentioned the idea of stealing GHC's test suite driver. This should definitely have all the features that we might need, including having different switches for each test, expecting tests to be broken and executing custom commands before a test is run. I spent some time looking into this and I fear that taking that driver out of GHC and adapting it to work with Links might be quite expensive. The driver relies on some Haskell code and this would have to be rewritten to OCaml.

On the other hand, I searched for various testing libraries for OCaml and didn't find anything that would offer the features needed for testing Links.

from links.

Another thing I would like to see in a test suite is some sort of summary at the end. Now I have to scan the results for failures, which is tiresome and error prone.

from links.

If anyone thinks that we should not merge this change please speak up before Monday the 24th (10 days from now). If there are no objections by that date I will merge.

from links.

Rebased my patch on top of Sam's fixes and refactored the code to eliminate repetitions. Awaiting feedback or any objections against merging.

from links.

Ugh. I have fixed the shredding part of the tests in the meantime, but with this change, my change is obsolete. It might even rebase cleanly?!

Anyway, looks good to me.

from links.

Merged and pushed.

I'm wondering whether to close this ticket or not. There's some useful discussion here about improving the testsuite and we might want to do this at some point in the future. I suggest to rename this issue and keep it open.

from links.

The problem named in the issue has been resolved, so I would suggest to close it and create a new issue about improving the test suite, collecting the useful discussion points there. Otherwise, no issue that generated interesting discussion would ever be closed.

from links.