e.g. https://github.com/haxiomic/console.hx

How to only print Failures info?

 1717 Assertions   1715 Success   2 Failures   0 Error

It's too difficult to find out Failures info in output logs.

Is possible only to print failure information?

It would be handy to have some means of ensuring that something compiles or doesn't compile in a test. Let's say I have a macro for validating my classes and I want to ensure that it does not compile:

class UnreasonableChecker {
    macro static public function build():Array<haxe.macro.Field> {
        var fields = Context.getBuildFields();
        for (field in fields) {
            if (field.name.startsWith('x') {
                Context.error('I HaTe vAriAblEs wHaT stARt WitH X', Context.currentPos());
            }
        }
        return fields
    }
}

@:build(UnreasonableChecker.build())
class NaughtyClass {
    private var x:Int;
    private var y:Int;
}

Could I in the context of a unittest ensure that this doesn't succeed in building?

When compiling test adapter sample project (https://github.com/vshaxe/haxe-test-adapter) with latest Haxe4 rc4 tink_testrunner sample produces following error output:

/usr/share/haxelib/tink_testrunner/0,7,2/src/tink/testrunner/Suite.hx:74: characters 29-33 : tink.testrunner.BasicSuite should be tink.testrunner.Suite
/usr/share/haxelib/tink_testrunner/0,7,2/src/tink/testrunner/Suite.hx:36: lines 36-42 : tink.testrunner.BasicSuite should be tink.testrunner.Suite
/usr/share/haxelib/tink_unittest/0,6,2/src/tink/unit/TestSuite.hx:18: characters 16-72 : tink.unit.TestSuiteBuilder0 should be tink.testrunner.Suite
/usr/share/haxelib/tink_unittest/0,6,2/src/tink/unit/TestSuite.hx:18: characters 16-72 : For function argument 'suites'
src/Main.hx:8: characters 14-102 : Called from macro here

Code like https://github.com/haxetink/tink_sql/blob/de2c6db5bb7e0b6f78ae358e8f6dba550082293c/tests/SubQueryTest.hx#L66-L80 could be rewritten as:

@:condition(this.driver.type == MySql)
public function anyFunc():Assertions {...}

Basically I'd like the possibility to be doing something like:

causeSideEffectA();
assert(X);
causeSideEffectB();
assert(Y);

I'm not even sure how this fits with async tests or anything - just opening the issue to see what ideas we can come up with.

Doesn't work with compilation server:
Type not found : tink.unit.Suite_0

@back2dos any idea if I have done anything wrong with the type cache?

In haxe3 it may produce a compiler stack overflow when if forces field typing during prepareBuffer.

Ideally it would be delayed, although that looks a little complicated. Haven't run into it in haxe4, so perhaps it's best to ignore it for the time being.

In the tink_sql tests equality between very long strings is tested. The results become hard to read because the full string gets printed. Maybe those should be truncate? Or not printed if the test is successful?

That will allow customizations on the output

Currently, abstracts are piped to Assert.stringify, which loses the type and just shows the runtime representation.

For @:exclude it should be easy. Just inject build macro and remove fields with such meta.

For @:include it might be tricky. We need to process all test classes too see if there are any @:include first, then exclude anything else.

Any ideas?

I've started working on a build macro based version that'll make the users write implements tink.unittest.TestSuite in the long run (I propose deprecating the current style for now). The main motivation was really to support the implementation of #17. Also, it'll allow making some transformations, like the one brought about by @:asserts.

So I would like to use the opportunity to see what we could do about some of the syntactic noise (e.g. having to have return asserts.done() in a fully synchronous test - which is the overwhelming majority of them - is a bit annoying) and simplifying things for newcomers.

I see two solutions to then express async tests:

1. Implicit: If the function returns a future, wait for it (also, if it returns a promise then report failure).
2. Explicit: Only tests decorated with @:async (for example) will be treated as such. In this case we have two options in turn:
   1. Pass a done:()->Void that allows terminating the test
   2. Expect a future in return. On first glance this seems nicer, because it makes the user return a single future rather than potentially calling done from various branches, but if the test is more easily written in such a way they'll just write return Future.irreversible(done -> ...) and have the same structure anyway. Also not having to learn about futures to write async tests has some appeal (this should not be decisive, but if both approaches are equally good from an experienced tink user's perspective, we can go for the one that's more approachable to newbies).

We could support all the solutions too, although I'm not sure that's practical. People gravitate towards consistency and as far as test suites go I can see the appeal of not having to figure out which style something is written in ^^

make sure the macro won't pick it up (and cause it to run) twice

https://github.com/haxetink/tink_unittest/blob/b0364ce/src/tink/unit/AssertionBuffer.hx#L19

• Support PosInfos
• implement optional argument code skipping manually (Context.typeof(description).getID() == 'haxe.PosInfos')

@:describe("GDC")
  @:case(32, 24, 8)
  @:case(42, 39, 3)
function testGdc(a:Int, b:Int, expected:Int) {
  return assert(gdc(a, b) == expected);
}

Or to use it for tests we already have:

@:describe
  @:case('http://example.com,foo.com/foo/bar', ['example.com', 'foo.com'])
  @:case('http://example.com:80,foo.com:81/foo/bar', ['example.com:80', 'foo.com:81'])
function testMultiIpv4(url:Url, hosts:Array<String>) {
  return 
    assert(url.host, hosts[0])
      .concat([for (i in 0...hosts.length) 
         assert(hosts[i] == url.hosts[i])
      ]);
}

Not sure about the particular syntax. Maybe it should even be part of the @:describe?

@:describe("multi host urls")
@:describe("  without port" ('http://example.com,foo.com/foo/bar', ['example.com', 'foo.com']))
@:describe("  with port" ('http://example.com:80,foo.com:81/foo/bar', ['example.com:80', 'foo.com:81'])

Or something like that ... the above would probably unnecessarily complicate treatment of @:describe. But some variation of this might do the trick, e.g @:case("description" (arg1, ... argN)) | @:case(arg1, ... argN).

The thing is, we have many tests where we're basically doing the same thing with just many different inputs. It would help us not only to separate the data and the actual test, but also avoid multiple asserts, which are still a little awkward as seen above.

I often need to write this:

task1()
  .next(function(result1) {
    asserts.assert(result1 == expected1);
    return task2();
  })
  .next(function(result2) {
    asserts.assert(result2 == expected2);
    return task3();
  })
  .next(function(result2) {
    asserts.assert(result2 == expected2);
    return Noise;
  })
  .handle(asserts.handle);
return asserts;

And the major problem here is that the promise and the actual assertions are separated, which added burden to reading.

I tried to "simplify" the syntax to something like this:

Promise.inSequence([
 assertAsync(lazy(() -> task1()), result1 -> asserts.assert(result1 == expected1)),
 assertAsync(lazy(() -> task2()), result2 -> asserts.assert(result2 == expected2)),
 assertAsync(lazy(() -> task3()), result3 -> asserts.assert(result3 == expected3)),
]).handle(asserts.handle);
return asserts;

But it still doesn't look very nice when the calls/assertions are multi-line.

Any suggestions on this?

Appendix:

An example use case:
task1 = create user
task2 = check user table should contain the user
task3 = check another table should contain some new rows etc

assert(v1 == 1 && v2 == 0) is now printed as (true == true)

it should instead be (1 == 1 && 0 == 0)

Also consider parenthesis...

Should handle things declared in super class

Not sure how this can fit into current architecture but it would be nice to consider a module with statics as tests since we can (in nightlies) define them without surrounding class.

// MyTest.hx
function test() 
  assert(true);

Roughly, this:

class TestBatch {
  public static macro function make(rest:Array<Expr>):Expr
}

For backward compatibility you probably want something like:

switch rest {
  case [macro [$a{exprs}]]: rest = exprs;
  default:
}