It would be great to have a way to 'stop' at the first error and make it the first thing on the page. This would be great for inspection on saucelab's failures.

The reason I ask this is because in my tests I've to do some CORS calls and it is easier to setup the origin to https://*.localtunnel.me in the backend service

Not sure if this is a good idea or not, but would there be any interest in exporting ZUUL_PORT? That way a static server is only a short line away:

server: "python -m SimpleHTTPServer $ZUUL_PORT"

Tried several different versions including latest, but always ends up looking like this:

> [email protected] test-stack /Volumes/Shared_HD/Users/eshao/omni
> DEBUG=zuul* node_modules/.bin/zuul test/hours/*.js test/*.js

  zuul control server active on port 65500 +0ms
  zuul bouncer active on port 65501 +1ms
  zuul:sauce add opera 9 Windows 2003 +0ms
  zuul:sauce add android 4.0 Linux +0ms
  zuul:sauce add iexplore 6 Windows 2003 +0ms
  zuul:sauce add iexplore 7 Windows 2003 +1ms
  zuul:sauce add iexplore 8 Windows 2008 +0ms
  zuul:sauce add iexplore 9 Windows 2008 +0ms
  zuul:sauce add iexplore 10 Windows 2008 +0ms
  zuul:sauce add iexplore 11 Windows 2012 R2 +0ms
  zuul:sauce add firefox 19 Linux +0ms
  zuul:sauce add safari 6 Mac 10.8 +0ms
  zuul:sauce add iphone 6.1 Mac 10.8 +0ms
  - testing: opera @ Windows XP: 9
  - testing: android @ Android: 4.0
  - testing: iexplore @ Windows XP: 6 7
  - testing: iexplore @ Windows 7: 8 9 10
  - testing: iexplore @ Windows 8.1: 11
  - testing: firefox @ Linux: 19
  - testing: safari @ OS X Mountain Lion: 6
  - testing: iphone @ iOS: 6.1
  zuul:tunnel tunnel requested to port 65501 +0ms
  zuul tunnel url https://gkdk.localtunnel.me/__zuul +3s
  zuul:sauce running opera 9 Windows 2003 +570ms
  - queuing: opera v9 (Windows 2003)
  zuul:sauce running android 4.0 Linux +4ms
  - queuing: android v4.0 (Linux)
  zuul:sauce running iexplore 6 Windows 2003 +1ms
  - queuing: iexplore v6 (Windows 2003)
  zuul:sauce open https://gkdk.localtunnel.me/__zuul +462ms
  - starting: opera v9 (Windows 2003)
cloud failure: Unexpected data in simpleCallback.

npm ERR! [email protected] test-stack: `DEBUG=zuul* node_modules/.bin/zuul test/hours/*.js test/*.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] test-stack script.
npm ERR! This is most likely a problem with the kites-omni package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     DEBUG=zuul* node_modules/.bin/zuul test/hours/*.js test/*.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls kites-omni
npm ERR! There is likely additional logging output above.
npm ERR! System Darwin 13.0.0
npm ERR! command "/usr/local/Cellar/node/0.10.22/bin/node" "/usr/local/bin/npm" "run" "test-stack"
npm ERR! cwd /Volumes/Shared_HD/Users/eshao/omni
npm ERR! node -v v0.10.22
npm ERR! npm -v 1.3.14
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /Volumes/Shared_HD/Users/eshao/omni/npm-debug.log
npm ERR! not ok code 0

i think for the next version of component, i'm going to build all the scripts + styles + tests into a .js and .css file, not including the relevant runner. how would i get that running with zuul?

• i see require()s in the example, so i'm guessing you browserify that stuff. the test.js will include its own require() implementation, so it shoulud be standalone minus the runner. WILL IT RUN!?
• CSS!?!?!?! though the use-case for that is small
• what if i packaged everything into a single .html file. can we do tests that way?

because i'm using my own module loader, i have my own require()s, but browserify is parsing those. thus, i get the terrible stack trace from #64

maybe the latest browserify fixes this? or maybe i'm doing somethin ghorribly wrong. i'm doing something like:

nlz build test.js
zuul --local 8080 --ui mocha-bdd -- build/test.js

• figure out how to copy file location(including lineno) to clipboard so it can quickly opened in chrome dev tools via CMD-O CMD-V Enter
• show the syntax highlighted code of the entire file in a pane to the right with the traced line highlighted when a more button is clicked

Example output. The unix return code also indicates success.

deshao-mbp ~/omni> node_modules/.bin/zuul --ui mocha-qunit test/hours/*.js test/*.js
  - testing: googlechrome @ Windows XP: 31
  - testing: safari @ OS X Mountain Lion: 6
  - testing: iexplore @ Windows XP: 6 7
  - testing: iexplore @ Windows 7: 8 9 10
  - testing: iexplore @ Windows 8.1: 11
  - queuing: googlechrome v31 (Windows 2003)
  - queuing: safari v6 (Mac 10.8)
  - queuing: iexplore v6 (Windows 2003)
  - starting: googlechrome v31 (Windows 2003)
  - starting: iexplore v6 (Windows 2003)
  - passed: googlechrome v31 (Windows 2003)
  - starting: safari v6 (Mac 10.8)
  - queuing: iexplore v7 (Windows 2003)
  - passed: safari v6 (Mac 10.8)
  - queuing: iexplore v8 (Windows 2008)
  - starting: iexplore v7 (Windows 2003)
  - starting: iexplore v8 (Windows 2008)
  - failed: iexplore v6 (Windows 2003): 36 failures
  - queuing: iexplore v9 (Windows 2008)
  - failed: iexplore v7 (Windows 2003): 33 failures
  - queuing: iexplore v10 (Windows 2008)
  - starting: iexplore v9 (Windows 2008)
  - starting: iexplore v10 (Windows 2008)
  - passed: iexplore v10 (Windows 2008)
  - passed: iexplore v9 (Windows 2008)
  - queuing: iexplore v11 (Windows 2012 R2)
  - failed: iexplore v8 (Windows 2008): 23 failures
  - starting: iexplore v11 (Windows 2012 R2)
  - passed: iexplore v11 (Windows 2012 R2)
  - all passed

Would be nice to parse my --slow etc. options.

Hi,

From my testing, iphone tests are starting and finishing (given the saucelabs interface) but zuul is never noticed of the end of the tests.

Zuul always says:

• starting: iphone v6.1 (Mac 10.8)

but the test never finishes. other browsers are ok.

https://github.com/vvo/lazyload has this problem so I removed the iphone line in zuul.yml

Looks like zuul will always enable checkLeaks when available - this isn't desired for me, since some libraries (Leaflet) use event handler globals that look to mocha like leaks. Could this become optional?

I'm using mocha as test interface and it has global detection turned on by default. My library fortress is actually introducing globals as part of it's tests. It would be nice if these testing frameworks's options could be set through the zuul.yml file.

Global leak introduced on firefox when run in saucelabs with a failing test. I have disabled it for now. Needs better testing and review before we re-enable it. The benefit is limited but breaking things is bad.

Noticed recently that with this config

ui: tape
browsers:
  - name: chrome
    version: 35..latest
  - name: firefox
    version: 30..latest

that Chrome tests are running on every possible platform. See:

This didn't used to happen before, and doesn't happen with Firefox tests. Maybe the browser json endpoint changed?

Probably use istanbul. Should be test framework agnostic.

I am receiving this failure quite often. I dug into it and put in some logging statements in the wd dependency in which it is happening, and here is the message:

  - starting: iphone v6.1 (Mac 10.8)
in simpleCB
undefined
ERROR Job url is not in progress. It may have recently finished, or experienced an error. You can learn more at https://saucelabs.com/jobs/url
cloud failure: Unexpected data in simpleCallback.

// zuul exits with error status at this point

In SauceLabs, it reports the test as: Client disconnected during getNewBrowserSession request.

I believe this occurs any time a given browser combination errors before it even gets to the part where mocha runs tests. For example, attempting to perform a console.log in IE when the console is closed.

I would suggest that the desired behavior would be to notify the user that the browser could not execute the test file due to pre-test evaluation errors and count it as a single failure instead of exiting the process.

opener will allow you to open the browser. I tried this:

"test-browser": "zuul --server 9000 test/tests.js && opener http://localhost:9000"

But it didn't seem to work since zuul takes over the console and won't let go.

I'm looking to setup a test suite to run using Sauce Labs. All specs run and pass when running locally at http://localhost:8080/__zuul. But when running on Sauce Labs, every browser results in one of the following two errors:

Error: timeout of 2000ms exceeded
    assertion https://steogwvfxz.localtunnel.me/__zuul/client.js:1118
    Anonymous function https://steogwvfxz.localtunnel.me/__zuul/client.js:691
    emit https://steogwvfxz.localtunnel.me/__zuul/framework.js:583
    fail https://steogwvfxz.localtunnel.me/__zuul/framework.js:4539
    failHook https://steogwvfxz.localtunnel.me/__zuul/framework.js:4564
    Anonymous function https://steogwvfxz.localtunnel.me/__zuul/framework.js:4603
    done https://steogwvfxz.localtunnel.me/__zuul/framework.js:4295
    Anonymous function https://steogwvfxz.localtunnel.me/__zuul/framework.js:4275

Error: timeout of 2000ms exceeded
    {anonymous}() https://guflvbqvub.localtunnel.me/__zuul/framework.js:4275

Had this been seen at all before?

Not so much an issue, and more like I need help! Really like how easy zuul is to use, but I am a little stuck on it's setup with superagent. Since superagent runs tests for node and the browser, the travis matrix ends up expanding to being like this:

• node 0.8
  • runs node tests
  • runs zuul tests
• node 0.10
  • runs node tests
  • runs zuul tests
• node 0.11
  • runs node tests
  • runs zuul tests

https://travis-ci.org/visionmedia/superagent/jobs/19407988

Any thoughts on how to flatten it out to run all the node tests, than each browser tests?

I'm loving zuul now that I'm finally diving in to try it out. One thing that kinda turns me off is that the currently recommended Travis integration setup:

1. Doesn't allow a good way to run the tests via node.js and browsers (dual-sided modules).
2. Shoves all different test output into one Travis "test". It would be ideal IMO if each browser was run in its own "test" on Travis.

What do we need to do to make this happen?

browserify

I'd like to provide an alternative way to create the browserify instance, which allows me to configure the following:

• transforms
• browserify opts (like ignore)
• adding shims

Additionally I'd like to provide the opts passed to .bundle(). This would alleviate the whole discussion around the --debug flag, since I could specify this option and more:

• debug, noparse, etc.

fixture

I'd also like to override the content of the index.html file (not just the path to it, but point it to a function that returns a string) in order to facilitate templating methods I use in my app for testing as well.

Of course when returning that html, I'd be responsible to include whatever is needed to make zuul and mocha work.

implementation

All this configuration could be provided via a simple JavaScript file to which I point zuul via a --config flag.

exports.browserify = function () {
  // create browserify here, add shims, transforms (but not files) and return it
  // when returned, zuul adds all test files and dependents
};

exports.bundleOpts = {}; // object to pass into `.bundle(opts)`

exports.fixture = function () {
  // return an html string here that you can create however you like
  // i.e. fs.readFileSync(myIndex) -- which may have extra content/script tags or whatever else I need
  // Handlebars.render(mytemplate, mycontext);
  // etc. -- opens up all possibilities needed
};

If any of these is not provided, zuul falls back to the defaults.

I'd like to use zuul to generate my tests, but my code is dependent on browserify transforms that I'd like to pass to zuul.

I think that I can add this to the CLI and then pass the options down through to the control_app, but I'd like to hear your thoughts on this.

I occasionally SauceLabs fails to spin up a browser (usually an OSX or iOS browser):

Error: The environment you requested was unavailable.: {"status": 13, "sessionId": "XXXXXX", "value": {"message": "Browser failed to start"}}

It would be nice if these were reported as "errors" in the stats, rather than printing out the full stacktrace to the terminal.

Given this project : https://github.com/vvo/in-viewport

ui: mocha-bdd
html: test/fixtures/template.html
scripts:
  - build/in-viewport.min.js
browsers:
  - name: chrome
    version: 29..latest
  - name: firefox
    version: 24..latest
  - name: ie
    version: 8..latest
  - name: iphone
    version: 5.0..latest
  - name: ipad
    version: 5.0..latest

When starting zuul locally:

zuul --local 8080 -- test/*.js

It hangs forever because it tries to load http://localhost:8080/build/in-viewport.min.js and does not find it.

[email protected] does not have this problem.

I constantly see this error when running zuul on travis. Seems to be ok when I run it locally. I reduced the concurrency so now tests run in serial but still no luck getting the full test suite to run start to finish.

Would it be a good idea to have glob functionnality in the scripts field? Like testling has.

So that I can write:

scripts:
  - "test/helpers/*.js"  

?

Their files have trailing commas (and maybe other issues).

The readme includes a screenshot of a test suite running with the Mocha HTML reporter:

I was wondering what the recommended way is to replicate this setup?

I have some browser-based tests that use a worker.js webworker source and a modalDialog.htm HTML that gets popped up in a modal dialog.

Trying to port these tests to Zuul doesn't seem to work since I get errors like "cannot find file /worker.js".

Hi, I switched to latest zuul on https://github.com/vvo/lazyload package.json which
has travis-ci.

The travis-ci job reports passed tests while the saucelab build says most tests failed.

What am I doing wrong?

Have any objections to updating browserify to 4.2.2 to pickup the security fix to syntax-error?

Hi, is there a way to setup mocha via some config-mocha.js script file listed at .zuul.yml?

I've seen there were some issue with the global leaks, I'm having those as well.

Using this.setup({ignoreLeaks: true}) or even this.globals(['myGlobVar*']) doesn't work since this inside the describe or it scopes refers to the TestSuite rather than the mocha.js instance. (I read #55 before)

So, I was hoping there is a way to setup mocha just once (not per test file) like

// config-mocha.js
mocha.timeout(25000);
mocha.globals(["jQuery*"]);

and...

# .zuul.yml
ui: "mocha-bdd"
scripts:
  - "/support/config-mocha.js"
  - "/support/jquery.js"

But for that to work, the framework script here: https://github.com/defunctzombie/zuul/blob/master/frameworks/index.html#L19

... should be moved up the scripts for-each loop so that the mocha var gets reachable.

Are there any side effects for moving that file include few lines up?

As title.

When running zuul on saucelabs, it does not report the 'status' of the pipeline anymore, and you do not know from the output which platform failed: https://travis-ci.org/mcollina/levelgraph#L3264. The tests on saucelabs are run.

Hey guys, I know, this issue popped up several times here (27, 10).

Looking at the commit's history @thlorenz 's --config option has gone now, and using

  "browserify": {
    "transform": ...
  }

in my package.json may not be always a good option. For example, I want to use brfs transform to test my package A, but it shouldn't be required when consumers of A are browserifying their bundle. Couldn't find any workaround yet.

Maybe I'm missing something?

Bad platform values result in

/Users/tmcw/node_modules/zuul/lib/flatten_browser.js:29
        avail.reduce(function(prev, curr, idx, arr) {

TypeError: Reduce of empty array with no initial value

This error could be improved to be more informative

Don't want the bundle size to grow out of control. It impacts test time and my bandwidth for localtunnel :)

Sauce Labs denotes between portrait and landscape orientation of mobile devices using the device-orientation option. They also denote between phone and tablet configurations of Android with the device-type option.

• device-orientation
  • tablet
  • portrait
• device-type
  • tablet

In the zuul.yml, it would be nice to configure one mobile browser that runs in both orientations, similar to how browser versions can be specified.

Just wanted to point out that the README is missing any form of a Ghostbusters reference, which made me make a face like this: ಥ_ಥ

Seems like when tests fail/misbehave, instead of getting properly test failures we get zuul problems.

We did not change the zuul version on our project, but certain tests started failing out of the blue:

With zuul 1.3.1 we get:

unknown line: "TypeError: 'undefined' is not an object (evaluating 'window.zuul_msg_bus.splice')"
failed to parse: TypeError: 'null' is not an object (evaluating 'msgs.forEach')
failed to parse:
failed to parse:   /Users/thlorenz/dev/cn/features/slideshow/node_modules/zuul/lib/phantom-run.js:26
unknown line: "TypeError: 'undefined' is not an object (evaluating 'window.zuul_msg_bus.splice')"
failed to parse: TypeError: 'null' is not an object (evaluating 'msgs.forEach')
failed to parse:
failed to parse:   /Users/thlorenz/dev/cn/features/slideshow/node_modules/zuul/lib/phantom-run.

which sends the tests in an endless loop. Same behavior in the browser.

With zuul 1.5.1 and 1.5.4 we get:

related slideshow data retrieval failed

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: failed to parse json: TypeError: 'null' is not an object (evaluating 'msgs.forEach')
    at Stream.<anonymous> (/Users/thlorenz/dev/cn/features/slideshow/node_modules/zuul/lib/PhantomBrowser.js:85:36)
    at Stream.EventEmitter.emit (events.js:95:17)
    at drain (/Users/thlorenz/dev/cn/features/slideshow/node_modules/zuul/node_modules/through/index.js:36:16)
    at Stream.stream.queue.stream.push (/Users/thlorenz/dev/cn/features/slideshow/node_modules/zuul/node_modules/through/index.js:45:5)
    at Stream.<anonymous> (/Users/thlorenz/dev/cn/features/slideshow/node_modules/zuul/node_modules/char-split/index.js:16:26)
    at Stream.stream.write (/Users/thlorenz/dev/cn/features/slideshow/node_modules/zuul/node_modules/through/index.js:26:11)
    at write (_stream_readable.js:583:24)
    at flow (_stream_readable.js:592:7)
    at Socket.pipeOnReadable (_stream_readable.js:624:5)

at least here it ends when run on the console (instead of going into an endless loop).
In the browser however it goes into an endless loop.

Endless Loop: all tests are rerun over and over.

Afaik some of the tests may be failing, but going into an endless loop like this is not the proper result. Also all errors trace to zuul which makes it impossible to figure out which of our tests are causing the problem and how.

Currently the latest browserify is v5 (and it has some new stuff), do you have any plans for updating?

I am using v1.5.4 of zuul.

- starting: <opera 12 on Windows 2003>

/home/ubuntu/kites-omni/node_modules/zuul/node_modules/localtunnel/client.js:75
            local.unpipe();
                  ^
TypeError: Object #<Socket> has no method 'unpipe'
    at Socket.remote_close (/home/ubuntu/kites-omni/node_modules/zuul/node_modules/localtunnel/client.js:75:19)
    at Socket.g (events.js:193:14)
    at Socket.EventEmitter.emit (events.js:123:20)
    at Socket._destroy.destroyed (net.js:357:10)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)

Related to localtunnel/localtunnel#28.

Can you please bump localtunnel version to 1.1.0 to fix this bug?

In .zuul.yml, there is no way to specify something like:

  2 browsers:
  3   - name: firefox
  4     version: 19,21

If you do something like:

  2 browsers:
  3   - name: firefox
  4     version: 19
  5   - name: firefox
  6     version: 21

It will yield the following and hang (not exit):

deshao-mbp (1)~/omni> npm run test-stack

> [email protected] test-stack /Volumes/Shared_HD/Users/eshao/omni
> DEBUG=zuul* node_modules/.bin/zuul test/hours/*.js test/*.js

  zuul control server active on port 60186 +0ms
  zuul bouncer active on port 60187 +1ms
TypeError: Cannot read property 'version' of undefined
    at /Volumes/Shared_HD/Users/eshao/omni/node_modules/zuul/lib/browsers.js:38:50
    at Array.reduce (native)
    at /Volumes/Shared_HD/Users/eshao/omni/node_modules/zuul/lib/browsers.js:37:19
    at Array.forEach (native)
    at /Volumes/Shared_HD/Users/eshao/omni/node_modules/zuul/lib/browsers.js:18:17
    at IncomingMessage.<anonymous> (/Volumes/Shared_HD/Users/eshao/omni/node_modules/zuul/lib/browser.js:30:17)
    at IncomingMessage.g (events.js:175:14)
    at IncomingMessage.EventEmitter.emit (events.js:117:20)
    at ClientRequest.<anonymous> (/Volumes/Shared_HD/Users/eshao/omni/node_modules/zuul/lib/browser.js:28:13)
    at ClientRequest.g (events.js:175:14)
    at ClientRequest.EventEmitter.emit (events.js:95:17)
    at HTTPParser.parserOnIncomingClient (http.js:1688:21)
    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:121:23)
    at CleartextStream.socketOnData (http.js:1583:20)
^C

This feature is useful for those on commercial Sauce accounts where minutes are limited. Often you'll want to test an old version of a browser (say, FF 19) and the latest one, but none of the ones in between.

Browserify and Mocha are pretty out of date.

Mocha should probably use peerDependencies to pick up whatever Mocha version you have installed already.

A similar argument could be made for browserify, IMO, but I admit it's less clear since in theory browserify's behavior shouldn't change that much between releases.

Ran into this while testing zuul-mp:
https://travis-ci.org/thlorenz/zuul-mp/jobs/14988284

> tap test/*.js
ok test/failing.js ...................................... 3/3
/home/travis/build/thlorenz/zuul-mp/node_modules/zuul/node_modules/superstack/index.js:140
        var listeners = self._events[event];
                                    ^
TypeError: Cannot read property 'drain' of null
    at httpSocketSetup (http.js:1749:10)
    at Server.connectionListener (http.js:1805:3)
    at Server.EventEmitter.emit (events.js:126:20)
    at TCP.onconnection (net.js:1051:8)
    at new Server (http.js:1767:8)
    at Object.exports.createServer (http.js:1776:10)
    at module.exports (/home/travis/build/thlorenz/zuul-mp/node_modules/zuul/node_modules/bouncy/index.js:24:16)
    at module.exports (/home/travis/build/thlorenz/zuul-mp/node_modules/zuul/lib/zuul.js:51:19)
    at Object.<anonymous> (/home/travis/build/thlorenz/zuul-mp/node_modules/zuul/bin/zuul:54:1)
    at Module._compile (module.js:449:26)
    at Object.Module._extensions..js (module.js:467:10)
    at Module.load (module.js:356:32)

Is there a way to force the OS? When you pick a browser version without OS, saucelabs seemingly randomly picks an OS (for example there's not guarantee that chrome 34 will run on OSX or Linux)

dafuq is dis?

I do a

$ npm cache clear
$ sudo npm install -g zuul 

it fails with

4275 error EEXIST, mkdir '/usr/lib/node_modules/zuul/node_modules/browserify/node_modules/umd/node_modules/rfile/node_modules/callsite'
File exists: /usr/lib/node_modules/zuul/node_modules/browserify/node_modules/umd/node_modules/rfile/node_modules/callsite                                               
Move it away, and try again.

I tried to include the full npm-debug.log, but couldn't submit it.