ciscoheat / buddy Goto Github PK

For each describe session, it seems that once the first test fails, the section tests stop being executed. This is very annoying when you introduce a big change - specially if your program takes a while to compile.

Thanks for the great work!

your changes doesn't work with swf8 target https://travis-ci.org/profelis/bindx2/jobs/41221109

Then fails to compile in Buddy.hx saying Invalid Package : src.blah.blah should be blah.blah

I tried taking the default HaxeFlixel template and integrating buddy tests with it, but I need some help. I added the Main.hx from the buddy readme to my project's root, and it runs fine from the command line. I then added this test to it (at the beginning of the constructor):

describe("A 100x100 rectangle at the origin", {
    var rect = new FlxRect( 0, 0, 100, 100 );
    it("should contain the point 50, 50", {
        rect.containsPoint( new FlxPoint( 50, 50 ) );
    });
});

I made sure the correct libraries were imported for FlxRect and FlxPoint, and added -lib flixel to the command for running the tests. I then get this error:

...\buddyhftest>haxe -lib buddy -lib flixel -main Main --interp
C:\HaxeToolkit\haxe\lib\flixel/git/flixel/math/FlxPoint.hx:3: characters 7-23 :
You cannot access the flash package while targeting neko (for flash.geom.Point)
Main.hx:2: characters 7-27 :     referenced here

Does this limitation mean that buddy isn't currently usable with a HaxeFlixel project?

Hi,

• please add some color to the terminal reporter.
• please allow comparing arrays with should.be (right now, it is by reference only)
  • it is usable with array.should.containExactly

I tried to do this:

try {
    #if utest
    Assert.results = new List<Assertation>();
    #end
    spec.run(done, status);
    if (!spec.async) done();
}
catch (e : Dynamic) {
    var stackTrace = CallStack.toString( CallStack.exceptionStack() );
    status(false, Std.string(e) + stackTrace);
}

But the async code swallows up the exception stack and I'm not familiar enough with promhx to know where to look next :)

Currently the progress bar looks like this:

.....PPPPXX.....P..X....

It would be nice if it could be split like:

Test1.hx: ....PP
Test2.hx: PPXX....
Test3.hx: .P..X....

I looked into the reporter code, that could be done with a custom reporter. But first we need the file name to be stored in the Spec class

Not sure how if the title correctly describes the issue but while trying to replicate a similar issue I'm seeing in Haxe 3.2 rc-1 I realized something like the following is failing in the 3.1.3.

 it("should pass", {
//test will pass if :Dynamic is removed
  var arr:Dynamic,
      fn;
  arr = new Array<String>();

  fn = function(){
    return arr;
  }
 trace(arr == fn()); //true
  arr.should.be(fn());
/*
throws:
 should pass (FAILED: Invalid call)
      @ /usr/lib/haxe/lib/buddy/0,17,0/buddy/internal/SuiteBuilder.hx:43
      @ /usr/lib/haxe/lib/buddy/0,17,0/buddy/internal/SuiteRunner.hx:179
*/
});

I'll try to post the 3.2 issue when I can although that may just be a bug in 3.2. It does throw the same error though.

I have 2 libs using buddy. The first one include the second one for its test using a -lib flag. When I run the tests I get the following errors:

/usr/lib/haxe/lib/buddy/0,9,0/buddy/Buddy.hx:3: characters 2-11 : Invalid package : test.unit should be unit

Both libs have a test/unit folder where the tests are. If I delete the tests from the second lib, everything works fine.

If a .should assertion fails, but the test still calls done() (in flash) then an AlreadyResolved error is thrown.

Perhaps the done() function could check the .isResolved value before resolving the promise again?

This is not really a buddy issue but since a library I use in js have its property accessor made to execute function, some of my test are failing when there result are printed on screen. Would there be a way to disable Std.toString in the output?

I use utest library with buddy. utest has this lines

https://github.com/fponticelli/utest/blob/d8ab198807434549f172dc18b157ec49327eb0cf/src/utest/Assert.hx#L289-L290

where keys() call without casting to StringMap or IntMap. haxe compiler (dce) remove keys() method as unused.

It's pretty useful once you started having lots of tests to only run a single one.
Could be used like so:

it.only('should run this test');

Thanks

I'm using Buddy primarily on a code base that targets Flash, so accept my twisted perspectives with that in mind. :)

I eventually want to get my Buddy specs running in Continous Integration, and I'm also a big fan of command line test runners.

The flash tests, though, need to run in the Flash Debugger (I use the open command on my Mac).

It'd be great if all the tests, no matter the target, reported their results back to a central command line test runner, who was then in charge of displaying the results in a common manner.

I've recently done some testing where my tests work against a node.js server running along side them. It seems to work well enough (once all the cross-origin stuff was worked out..) Perhaps a test runner could start a server and then launch the individual target's test runners. Those 'child' runners would communicate their results back to the server on a well-known port which would then handle the job of actually displaying the results on screen (or writing them to file in the JUnit-style for CI!)

In my specific use case this will help me trace the residue clues about the causes of the failure.
Because my tests are environment-aware and the environment resets at the beginning of each test.

It would be handy to have this check, just using Std.is(). And obviously the negative too: should.not.beType().

I'm not precious about the exact syntax or wording if you think of something better :)

I have the following test:

before(function () {
  //initialize code
})

describe('#function', function () {
   it('should do stuff');
});

describe('#anotherFunction', function () {
  it('should do other stuff');
});

The initialisation code in before seems to not be called before each 'it', but only before each 'describe'. If I duplicate the before in each describe it works fine.

Is it the normal behaviour ? In mocha there is before and beforeEach, maybe that's what is missing ?

By the way, I've started using utest.Assert and it works great, thanks !

Right now, the only way to mark a test as not pending is to actually assert.

Sometimes, I just want to see that my callback happened...

I have interest in making async tests work on PHP.

I understand that PHP doesn't support true async, but my use case is:

• I have some code that I want to compile to Neko, PHP and JS
• Because of JS, I'm using async style code, and Neko and PHP just "pretend"
• They use async method signiatures, but actually run synchronously...

Anyway, this is all fine, but the async tests don't work because promhx doesn't support PHP.

So my question: what would be the best approach to support pseudo-async PHP tests in Buddy:

1. Work around the need for promhx on PHP
2. Go ask @jdonaldson how to make Promhx work on PHP?

it("should", function(done)done()); this statement cause following compile-time bug

/usr/lib/haxelib/promhx/1,0,21/src/main/promhx/base/AsyncBase.hx:167: characters 16-21 : Can't create thread from within a macro
/usr/lib/haxelib/promhx/1,0,21/src/main/promhx/base/AsyncBase.hx:186: characters 20-40 : Called from
/usr/lib/haxelib/promhx/1,0,21/src/main/promhx/base/EventLoop.hx:60: characters 24-28 : Called from
/usr/lib/haxelib/promhx/1,0,21/src/main/promhx/base/EventLoop.hx:77: characters 12-15 : Called from
/usr/lib/haxelib/promhx/1,0,21/src/main/promhx/base/AsyncBase.hx:176: lines 176-188 : Called from
/usr/lib/haxelib/promhx/1,0,21/src/main/promhx/Promise.hx:120: characters 7-26 : Called from
/usr/lib/haxelib/promhx/1,0,21/src/main/promhx/base/AsyncBase.hx:256: characters 30-49 : Called from
/usr/lib/haxelib/promhx/1,0,21/src/main/promhx/Promise.hx:102: characters 8-36 : Called from
/usr/lib/haxelib/buddy/0,18,1/buddy/SuitesRunner.hx:31: lines 31-43 : Called from
/usr/lib/haxelib/buddy/0,18,1/buddy/GenerateMain.hx:195: characters 4-16 : Called from
?:1: characters 1048575-8796094070782 : Called from
Aborted
Build failed

I can break this out into separate issues if you like but I figured I'd just give a summary of what I've seen first

• Initial run gave me /usr/local/lib/haxe/lib/buddy/1,0,1/buddy/internal/GenerateMain.hx:179: characters 3-36 : Type not found : SuitesRunner
  • Importing this into my main class fixed this.
• For CPP I'm getting /usr/local/lib/haxe/lib/buddy/1,0,1/buddy/BuddySuite.hx:12: characters 7-22 : Type not found : hxcpp.StaticStd
• For Neko I used to be able to run the tests using --interp this doesn't seem to work anymore
• For js, neko & php I'm getting test failures that used to pass. I'm pretty sure they weren't false positives in the previous version but I'll investigate further on my end to be sure.

In SuitesRunner.hx the following function is defined.

private function isSync(funcs : Iterable) : Bool {
for (f in funcs) switch f {
case Async(): return false;
case Sync():
}
return true;
}

On my Windows Server 2008 machine, this gets turned into the below in python:

def isSync(self,funcs):
# C:\HaxeToolkit\haxe\lib\buddy/2,2,0/buddy/SuitesRunner.hx:306
# C:\HaxeToolkit\haxe\lib\buddy/2,2,0/buddy/SuitesRunner.hx:306
tmp = HxOverrides.iterator(funcs)
while tmp.hasNext():
if ((tmp.next().index) == 0):
return False
elif ((tmp.next().index) == 1):
pass
else:
pass
# C:\HaxeToolkit\haxe\lib\buddy/2,2,0/buddy/SuitesRunner.hx:310
return True

When run I get this error do to the multiple tmp.next() calls:

File "C:\Users\administrator.HOSTINTEGRATION\Desktop\Haxe\solidfire-sdk-haxe\bin\RunTests.py", line 8040, in isSync
elif ((tmp.next().index) == 1):
AttributeError: 'NoneType' object has no attribute 'index'

Updating the code to the following fixes the issue:

private function isSync(funcs : Iterable) : Bool {
for (f in funcs) switch f {
case Async(_): return false;
default:
}
return true;
}

This does not happen on my Windows 10 PC or on Mac.

Using Python 3.5 and Buddy 2.2.0, and Haxe 3.2 on all machines.

I will open a bug with the haxe project, but I wasnt sure if you wanted to work around the issue in the mean time.

To reproduce:

class Main {

    static function main() {
    var reporter = new ConsoleReporter();
    var runner = new SuitesRunner([new Test()], reporter);
    runner.run();
    }
}

class Test extends BuddySuite {

  public function new():Void {
    var x = 1;

    describe("test one", {

      it("fail", {
        throw "fail";
      });

      it("pass", {
        x.should.be(1);
      });

    });
  }
}

Neko output (correct):

X.
test one
  fail (FAILED: fail)
    @ Main.hx:26
  pass (Passed)
2 specs, 1 failures, 0 pending

C# output (missing report of second step):

X.
test one
  fail (FAILED: fail)

Unhandled Exception: System.NullReferenceException: Object reference not set to
an instance of an object.
   at promhx.base.AsyncBase__handleError_159__Fun`1.__hx_invoke1_o(Double __fn_f
loat1, Object __fn_dyn1)
   at promhx.base.AsyncBase__handleError_176__Fun`1.__hx_invoke0_o()
   at promhx.base.EventLoop.f()
   at promhx.base.AsyncBase`1.immediateLinkUpdate[A,B](AsyncBase`1 current, Asyn
cBase`1 next, Function f)
   at promhx.Promise`1.then[A](Function f)
   at buddy.SuitesRunner.run()
   at EntryPoint__Main.Main()

Note that wrapping it in a try/catch will not solve the issue, but only prevent the stack trace to be logged.

Full call stack here

https://travis-ci.org/profelis/bindx2/jobs/71731758

Source

https://github.com/profelis/bindx2/blob/master/test/Tests.hx

Looks like main problem in promhx library.

Hi,
is unit testing macros supported ?
If it is, how would you go about it ?

Thanks

I'm trying to hack buddy, so I manually cloned it and haxelib dev buddy path/to/buddy.
Try it with the following:

import buddy.*;
using buddy.Should;

class Test extends BuddySuite implements Buddy {
    public function new():Void {

    }
}

compilation error:

/Users/andy/Documents/workspace/buddy/src/buddy/tests/AllTests.hx:383: characters 4-17 : Unknown identifier : Assert
/Users/andy/Documents/workspace/buddy/src/buddy/tests/AllTests.hx:384: characters 4-15 : Unknown identifier : Assert
/Users/andy/Documents/workspace/buddy/src/buddy/tests/AllTests.hx:389: characters 4-17 : Unknown identifier : Assert
/Users/andy/Documents/workspace/buddy/src/buddy/tests/AllTests.hx:395: characters 5-17 : Unknown identifier : Assert

The interesting thing is if I disable the dev version, and use version 0.11.1 instead, it compiles fine.
Any idea?

The readme gives the impression that is does but I can't seem to get it to work.
@exclude seems to work everywhere.

package;

import buddy.*;
using buddy.Should;

class RunTests extends SingleSuite
{

    public function new() {
        for(folder in sys.FileSystem.readDirectory('nonexistent')) {
            describe(folder, {
                it('should be $folder', {
                    true.should.be(true);
                });
            });
        }

    }
}

The above code on nodejs fails with:

/Users/kevin/Codes/test/bin/node/tests.js:254
        runner.haveUnrecoverableError(err);
              ^

TypeError: Cannot read property 'haveUnrecoverableError' of null
    at process.<anonymous> (/Users/kevin/Codes/test/bin/node/tests.js:254:8)
    at emitOne (events.js:77:13)
    at process.emit (events.js:169:7)
    at process._fatalException (node.js:211:26)

which is not meaningful.

Running the same code on neko gives you some hint about the error of reading a non-existent directory though.

Getting this. It used to work, some test I added seems to have caused it, but I've not been able to hunt it down. Works fine in flash.

...Called from ? line 1
Called from buddy/GenerateMain.hx line 117
Called from buddy/SuitesRunner.hx line 30
Called from promhx/Promise.hx line 102
Called from promhx/base/AsyncBase.hx line 221
Called from promhx/base/AsyncBase.hx line 148
Called from promhx/base/EventLoop.hx line 73
Called from promhx/base/EventLoop.hx line 57
Called from promhx/base/AsyncBase.hx line 158
Called from promhx/base/AsyncBase.hx line 145
Uncaught exception - Invalid field access : length

Any guidance as to what to look for?

-dce full breaks tests

node js_test.js

/home/as3boyan/HIDE-atom-shell/bin/js_test.js:166
            var $it1 = b.suites.iterator();
                                ^
TypeError: Cannot call method 'iterator' of undefined
    at includeMode (/home/as3boyan/HIDE-atom-shell/bin/js_test.js:166:24)
    at Object.buddy.SuitesRunner (/home/as3boyan/HIDE-atom-shell/bin/js_test.js:174:3)
    at Function.TestMain.main (/home/as3boyan/HIDE-atom-shell/bin/js_test.js:92:15)
    at /home/as3boyan/HIDE-atom-shell/bin/js_test.js:882:10
    at Object.<anonymous> (/home/as3boyan/HIDE-atom-shell/bin/js_test.js:883:3)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)

Hi,
I've installed buddy from haxelib, written the README example and ran it with:

  haxe -lib buddy -main Main --interp  

I get the following error:

/usr/lib/haxe/lib/buddy/0,1,0/buddy/Buddy.hx:3: characters 2-11 : Invalid package : lib.utest.compat.unit should be unit

I don't know where this is coming from, the only haxelib using utest I have installed seems to be promhx.

Thanks !

It appears that binding affects the order parameters are passed into a function in python when you have optional parameters and a PosInfos param.

Here is some example code to show the problematic behavior:

import buddy.*;
using buddy.Should;
using haxe.PosInfos;

class Sandbox extends BuddySuite
{
    public function new()
    {
        describe("Bind",
        {
            it("should bind properly",{

                Sys.println("calling without bind");
                showBindProblem();

                Sys.println("\ncalling with bind");
                showBindProblem.bind().should.throwAnything();

                Sys.println("\ncalling with bind and an empty array");
                showBindProblem.bind([]).should.throwAnything();

            });
        });
    }

    private function showBindProblem(?args:Array<Dynamic>,?p:haxe.PosInfos)
    {
        Sys.println("args: " + args);
        Sys.println("p: " + p);
    }
}

This produces the following output:

calling without bind
args: null
p: { className : Sandbox, fileName : Sandbox.hx, lineNumber : 14, methodName : new }

calling with bind
args: { className : Sandbox, fileName : Sandbox.hx, lineNumber : 17, methodName : new }
p: null

calling with bind and an empty array
args: []
p: { className : Sandbox, fileName : Sandbox.hx, lineNumber : 20, methodName : new }

I'm writing more tests with buddy at the moment and like it a lot !
One useful feature that I use in mocha is the "only" function, described here:
http://mochajs.org/#exclusive-tests

This allows to focus on only one test or test suite while debugging.
If you think it can make a good addition, I can try implementing it.

Thanks

I am running buddy using 10+ test "Suites". Each suite has over 20+ "Describes" containing 10+ "Its".

After the 30-35th testcase runs (30-35th dot is output) I get a StackOverflow Exception when targeting C#.

I added a printout for the callstack using CallStack.callStack().length in SuitesRunner.hx -> forEachSeries-> inside the next() function. It reaches well over 3,500 when the crash occurs. It looks like there are too many nested method calls in SuitesRunner.hx when you are running a lot of testcases in one batch.

Also, version 2.0.1 seemed stable for Java and Python, but 2.0.2 is causing failures for me in both languages.

Please let me know if you need any additional information.

I've started to use Buddy for a new project, it's great so far !

One of the assertions I use when doing BDD in JS is "equal" or "eql", comparing 2 objects recursively by value.
Is it something which can be done ?

Thanks !

It'd be nice to be able to wire it right up as an error handler on promises and callbacks.

Then the test would fail with a nice error message when the error callback is called.

Something like this:

            it("should make a GET request with promise", function(done, err) {
                var p = c.getJson("/hello_world");
                p.then(function(r) {
                    r.statusCode.should.be(200);
                    r.data.title.should.be('Hello World');
                    done();
                });
                p.catchError(err);  // <-- This would fail the test with a message as passed to err()
            });

Will be great to add custom EvalTest for buddy in https://github.com/waneck/travis-hx library

default EvalTest
https://github.com/waneck/travis-hx/blob/master/extra/evaluate-test/EvalTest.hx

I'm trying to test buddy on an existing project but [this line](https://github.com/ciscoheat/buddy/blob/master/src/buddy/tools/AutoIncluder.hx#L39] results in hundreds of compiler errors. This includes things like:

• "neko" / "php" packages being entered into when that is not the current platform. (3rd party code, not the current platform).
• 3rd party libraries with broken but unreferenced code causing compiler errors. (In my case, an old version of thx that I am still using).

etc. Because these errors occur at macro time they are particularly difficult to debug.

Can we change the build macro to have an optional "includePackages" argument? The usage would remain the same if you just use implements Buddy, but for poor souls such as myself we could call the build macro manually with just the package we wish to include.

I'm happy to create a pull request if you are okay with it.

A common pattern when working with nested describes is to do common configuration once in a before on the outer describe.

In buddy, though, the befores only seem to apply to the it calls directly within the describe.

It'd be nice if there was a stack of befores that would execute in the order of outermost first.

Not sure if this can be solved with macro but it would be great to be able to use should (even if value is null at runtime)

for example the following always fails

err.should.be(null)

if err is null you get

GenerateMain.hx:182: TypeError: Cannot read property 'should' of null

else you get an test error as err is not null

Is it possible to run next test class only when all the async tests of current class are finished?

I am setting up many tests (about 60) that are pending for now, but will be async tests once implemented.

// PENDING:
it("Should send an email to that teacher, with a pre-approved invitation", function () {});

// PASS:
it("Should send an email to that teacher, with a pre-approved invitation", function (done) { done(); });

I feel these should both be pending. Do you agree?

It would be great if we could use before() and after() only once before and after a test suite.

beforeAll/afterAll or some other method name would work exactly like actual before() and after() but just once.

Thanks for considering.

see: https://travis-ci.org/kevinresol/tink_remoting/jobs/126115012#L1772-L1775

It's a little unexpected that buddy 1.0+ now runs the tests on threads by default. It becomes harder to debug, and furthermore, there is some testing that needs to be done on the main thread, but I can't figure out how to disable threading. Is there any way to do it?

libs: -lib tink_core -lib buddy

Main.hx

import buddy.*;

class Main implements Buddy<[
    Test,
]> {}

Test.hx

import buddy.*;
using tink.CoreApi;

class Test extends BuddySuite {
    public function new() {
        Future.ofMany([
            Future.sync(Success(Noise)),
            Future.sync(Success(Noise)),
        ]);
        describe('Test', {
            it('dummy', {});
        });
    }
}

The above code fails with:

/Users/kevin/Development/haxelib/tink_core/git/src/tink/core/Future.hx:77: characters 18-30 : tink.core.Future<Array<ofMany.A>> should be tink.core.Future<A
rray<fromMany.A>>
/Users/kevin/Development/haxelib/tink_core/git/src/tink/core/Future.hx:77: characters 18-30 : Type parameters are invariant
/Users/kevin/Development/haxelib/tink_core/git/src/tink/core/Future.hx:77: characters 18-30 : ofMany.A should be fromMany.A
/Users/kevin/Development/haxelib/tink_core/git/src/tink/core/Future.hx:78: characters 11-14 : tink.core.Future<Array<ofMany.A>> should be tink.core.Future<A
rray<fromMany.A>>
/Users/kevin/Development/haxelib/tink_core/git/src/tink/core/Future.hx:78: characters 11-14 : Type parameters are invariant
/Users/kevin/Development/haxelib/tink_core/git/src/tink/core/Future.hx:78: characters 11-14 : ofMany.A should be fromMany.A

Make sure not to put the two classes in a same file, otherwise it will work.
Not sure if it is Haxe problem or buddy's or tink_core's problem...

cc @back2dos

after upgrading to 0.12.0 an error gets thrown when I try to compile and run:

output:
usr/lib/haxe/lib/buddy/0,12,0/buddy/BuddySuite.hx:202: characters 33-53 : Invalid field access : include
/usr/lib/haxe/lib/buddy/0,12,0/buddy/BuddySuite.hx:153: characters 2-43 : Called from
test/TestMain.hx:10: lines 10-14 : Called from
:1: character 0 : Called from
test/TestMain.hx:8: lines 8-16 : Called from
:1: character 0 : Called from
/usr/lib/haxe/std/neko/_std/Type.hx:108: characters 9-25 : Called from
/usr/lib/haxe/lib/buddy/0,12,0/buddy/GenerateMain.hx:111: characters 17-62 : Called from
?:1: characters 1048575-8796094070782 : Called from
Aborted
[Finished in 0.5s with exit code 1]

error https://travis-ci.org/waneck/travis-hx/jobs/68144580#L459

Test source code https://github.com/profelis/travis-hx/blob/patch-1/tests/fail/BuddyTests.hx

And then display them :)