This is a two-part feature (and neither makes much sense without the other.)

1. Add something like this to the output: Fuzzing with initial seed 1234 (Not sure if it should go at the beginning or at the end, or what the exact wording should be.)
2. Accept a CLI flag for what numeric seed to use on init.

This is no substitute for copying out the inputs of fuzz tests into failing unit tests, but would let you give the seed to a coworker so they could quickly reproduce exactly the set of failures you saw.

You can look at the master branch in elm-test and see the new API. I will release it soon as elm-test version 3.0.0, so I figure this repo should be updated as well. Once it's in alignment, I'll add a reference to here in my documentation (basically I'll accept this pull request)

This issue is more for the Googlers than for you, Richard.

I was using Hyper.app as my terminal on macOS Sierra, with a custom theme installed.
All negative test output was getting hidden from my terminal. Tests would fail, and none of the info about the tests would appear. I eventually figured out that my terminal was treating the control characters used to dim the output text (chalk.red.dim(..)) were being interpreted as commands to clear the text buffer.

I almost filed a bug with Hyper.app, but when I upgraded my config file, and used a different theme, the problem disappeared. Everything started working famously again.

So there you go, Googlers.

See https://gist.github.com/fredcy/e4d9a37e621f881057339dbf87144aa8 for SSCCE.

We should be able to output failures in a JUnit XML format, so CI servers can read it.

Notes:

• Test results should include duration, so we'll need to measure that
• It appears Jenkins now supports nested <testsuite>s, even though I believe they are not officially supported in the JUnit XML spec.

npm test runs ci.sh, which is a Bash script that works great on OS X and Linux (via Travis), but which segfaults on Windows (via AppVeyor).

It would be great to port ci.sh to Node and run it via npm test instead of using a Bash script, so we can confirm that things are working on Windows.

I am just a clueless user.

[email protected] postinstall /usr/local/lib/node_modules/elm-test/node_modules/node-elm-compiler/node_modules/elm
node lib/install.js

✖ No binary found matching your system. It's probably not supported.
✖ elm pre-build test failed
✖ No binary found matching your system. It's probably not supported.
✖ elm-doc pre-build test failed
✖ No binary found matching your system. It's probably not supported.
✖ elm-make pre-build test failed
✖ No binary found matching your system. It's probably not supported.
✖ elm-package pre-build test failed
✖ No binary found matching your system. It's probably not supported.
✖ elm-reactor pre-build test failed
✖ No binary found matching your system. It's probably not supported.
✖ elm-repl pre-build test failed

/usr/local/bin/elm-test -> /usr/local/lib/node_modules/elm-test/bin/elm-test
[email protected] /usr/local/lib/node_modules/elm-test
├── [email protected]
├── [email protected] ([email protected])
└── [email protected] ([email protected])

When using elm-check with elm-test, the last argument to Check.Test.test is a Random.Seed. If running the tests inside a custom handler one could do something like

seed : Random.Seed
seed = Random.initialSeed initialSeed

port initialSeed : Int

in the Main module and then do

Elm.worker( Elm.Main, { initialSeed: Date.now() } )

to run the tests with a random initial seed value. It would be nice to be able to do something similar when using the elm-test package. Currently the initial value can't be set randomly so every run is the same. I imagine this would have to done thorough a flag so that it wouldn't require the port to be declared. So the API could for example be:

elm-test TestRunner.elm // normal test suite, no port
elm-test --seed TestRunner.elm // would require a seed port, pass in current time
elm-test --seed=1234 TestRunner.elm // allow passing in a seed value to recreate failing run

Thoughts?

Travis is fine, but AppVeyor hangs. No idea what the problem might be.

https://ci.appveyor.com/project/rtfeldman/node-test-runner/build/job/f985x4rif8t3tad9

It just stops printing output here:

It would be nice to support interactive testing where the runner watches loaded files for changes and reruns the tests when the files change. I think this would be particularly nice if it was optimized for speed so that once you hit save in the editor you could see your tests finished by the time you switch to the console. To achieve this it would make sense to build these optimizations:

1. Work out the minimal subset of tests that need to run on a change. Given the following project:

   Main.elm imports A.elm and B.elm. B.elm imports C.elm. Each of these has a corresponding test file that only imports the file it tests. When I make a change to B.elm, I will only need to run tests from BTests.elm and MainTests.elm, since only these can have possibly changed.

   For this to work you would need to get the import tree from the compiler, which I don't know if it supported. When the dependency tree changes, it would make sense to re-run the entire suite.

2. Only run perhaps 10 fuzz tests rather than a hundred. Since you will run tests on every save, they should get decent coverage. You could perhaps run 100 on the first run.

Of course #51 would fit very well into this proposal as well.

Jacob on Slack had a cool idea, and I'm just gonna screencap it here:

Here's a design for how this could work:

1. We now have the guarantee that each test has a unique description.
2. We could use this as a unique identifier, and serialize the outcome of all tests to disk after a run. (e.g. {"id": ["top-level describe", "nested describe", "my description"], "passed": true}.)
3. The next time we detect a watch change, we could diff the new tree of tests against the old one before running. Any test that had passed: true last time, we don't bother re-running this time. (This way, we automatically add any new tests to the mix.)
4. If the tests we re-ran all passed, then we can go back and re-run all the others to make sure they still pass.
5. Profit!

If deps are missing currently the user gets an error message and has to work out how to resolve the issue themselves. It seems we could do this for them and avoid this workflow bump entirely.

In my project using previous version of elm-test I had

test/unit/
test/integration

I was using elm-make TestRunner.elm --output tests.js && node tests.js to run the unit test.

I'm trying to upgrade to elm-test 2 and node-test-runner. But it seems I need to put my test runner in tests/Main.elm.

I would like to put my Main file in tests/unit/Main.elm. Is there an option for this?

I have a simple test in my project that crashes node-elm-test. To reproduce, do

git clone https://github.com/akeeton/nickel-havana.git
cd nickel-havana
git reset --hard f62db33
cd test
elm-test src/TestMyList.elm

The output I get is

Compiling src/TestMyList.elm
Some new packages are needed. Here is the upgrade plan.

  Install:
    deadfoxygrandpa/Elm-Test 1.0.4
    elm-lang/core 2.1.0
    maxsnew/IO 1.0.1

Do you approve of this plan? (y/n) 
y
Downloading deadfoxygrandpa/Elm-Test
Downloading elm-lang/core
Downloading maxsnew/IO
Packages configured successfully!
Success! Compiled 41 modules.
Successfully generated /tmp/elm_test_11571-14917-b21vna

Successfully compiled src/TestMyList.elm
Running tests...
Cannot read property 'make' of undefined
[stdin]:8147
                throw error;
                      ^
TypeError: Cannot read property 'make' of undefined
    at init ([stdin]:8134:20)
    at Object.Elm.worker ([stdin]:8032:11)
    at [stdin]:11615:22
    at [stdin]:11653:3
    at Object.exports.runInThisContext (vm.js:74:17)
    at Object.<anonymous> ([stdin]-wrapper:6:22)
    at Module._compile (module.js:460:26)
    at evalScript (node.js:431:25)
    at Socket.<anonymous> (node.js:164:11)
    at Socket.emit (events.js:129:20)

As a control, I have added your example test.

elm-test src/NodeTestExampleTest.elm

It works as expected:

Compiling src/NodeTestExampleTest.elm
Success! Compiled 1 modules.
Successfully generated /tmp/elm_test_11571-15007-1a36lod

Successfully compiled src/NodeTestExampleTest.elm
Running tests...
  1 suites run, containing 3 tests
  0 suites and 2 tests passed
  1 suites and 1 tests failed

Test Suite: A Test Suite: FAILED
  Addition: passed.
  String.left: passed.
  This test should fail: FAILED. not True

Once I added an import of my own source file, the compile of TestRunner failed. It didn't find the module Css (my project uses elm-css).

My workaround is: mv tests test because then node-elm-test doesn't cd to tests, and then it can find the installed modules in elm-stuff.

I also added test as a source directory in elm-package.json. (And I had to manually install elm-test and elm-console packages, because init didn't add them to elm-package.json, but that's probably a separate issue)

Because of the way that we're structuring our code alongside other components, we would prefer not to use the nested elm-package.json style that elm-test uses by default. It would be great if the elm-test runner was able to run tests from modules with other names.

Using latest elm-test runner, the runner crashes with this error:

$ elm-test TestRunner.elm
Success! Compiled 1 module.
Successfully generated /tmp/elm_test_116719-15843-1kmv9mo.edvilcv7vi.js
undefined:1929
                        throw new Error(
                        ^

Error: You are giving module `Main` an argument in JavaScript.
This module does not take arguments though! You probably need to change the
initialization code to something like `Elm.Main.fullscreen()`
    at init (eval at <anonymous> (/usr/lib/node_modules/elm-test/bin/elm-test:86:37), <anonymous>:1929:10)
    at Object.eval [as callback] (eval at <anonymous> (/usr/lib/node_modules/elm-test/bin/elm-test:86:37), <anonymous>:1973:17)
    at step (eval at <anonymous> (/usr/lib/node_modules/elm-test/bin/elm-test:86:37), <anonymous>:2613:39)
    at Timeout.work [as _onTimeout] (eval at <anonymous> (/usr/lib/node_modules/elm-test/bin/elm-test:86:37), <anonymous>:2671:15)
    at tryOnTimeout (timers.js:232:11)
    at Timer.listOnTimeout (timers.js:202:5)

TestRunner.elm looks like this:

port module Main exposing (..)

import Test.Runner.Node exposing (run)
import Tests
import Json.Encode exposing (Value)


main : Program Never
main =
    run emit Tests.all


port emit : ( String, Value ) -> Cmd msg

$ elm --version
0.17.1

$ elm-test --version
0.17.3

$ node --version
v6.4.0

For example: https://travis-ci.org/rtfeldman/node-test-runner/jobs/179291777

The README for elm-test states that one needs to manually copy dependencies from the root elm-package.json to the test copy. Node is fully capable of reading files and parsing JSON so it would be a nice enhancement if running elm test init would handle this automatically. It could also do a similar check before running tests to keep the files up to date.

Hello!

I notice that this tool now generates a Main module when running init. This seems very likely to collide with existing modules in many projects. Perhaps we could instead generate TestMain ?

From elm-discuss:

I'm running Ubuntu 14.04. It installs OK I think,. Then (for example)

john@john:~/Development/elm/elm-binary-base64/test$ elm-test check/B64Check.elm
Exception thrown when attempting to run Elm compiler "elm-make":
TypeError: Incorrect value for stdio stream: inherit

or:

john@john:~/Development/elm/elm-binary-base64/test$ elm-test init
elm-package.json already exists

/usr/local/lib/node_modules/elm-test/bin/elm-test:41
childProcess.execSync(elm["elm-package"] + " " + command + elmOptions, {
^
TypeError: Object # has no method 'execSync'
at execElmPackageSync (/usr/local/lib/node_modules/elm-test/bin/elm-test:41:20)
at Object. (/usr/local/lib/node_modules/elm-test/bin/elm-test:45:3)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Function.Module.runMain (module.js:497:10)
at startup (node.js:119:16)
at node.js:902:3

(not at all sure what I'm doing here)

Hello!

Trying to use elm-test without the Elm platform installed globally results in an error like so:

$ ./node_modules/.bin/elm-test
Could not find Elm compiler "elm-make". Is it installed?

Could we run elm-make from node_modules if it is present there? This would make it easier to work with multiple versions of Elm.

I'm using elm-history which is wrapping window.location (e.g. this), so when running a test via console we get:

Successfully compiled ./src/elm/TestRunner.elm
Running tests...
Cannot read property 'pathname' of undefined
[stdin]:11129
                throw error;
                      ^
TypeError: Cannot read property 'pathname' of undefined
    at Object.Elm.Native.History.make ([stdin]:9870:54)
    ...

Any experience/ pointers on how to mock this data?

go test is an awesome feature of Go where the Go distribution ships with a unit testing feature. Running go test on a project will automatically find files with <x>_test.go and run them. It gives a clear output of failed cases and successful files, along with a benchmark for how long it ran for.

Not quite sure how this maps over to elm-test just yet, but the desired goal is to make it so that elm-test can be used with minimal set up.

Some thoughts

@dgtized suggested leiningen's profiles as a possible solution. Ever since he suggested I can't help but feel is the right way forward. The gist of profiles is that you could have a root elm-package.json which would be a subset of a test/elm-package.json, though with a modified sources. This is already the case for every test I've ever seen in Elm. This would avoid having two different elm-package.json files that need updating each time the deps need updating, and improve the ability for us to inject deps. For example, by default we could inject elm-test or elm-check with io into a test project, allowing us to use elm-test just like go test

Make sure any applicable issues have been migrated from project-fuzzball/test#16 to here.

would it be possible to compile the tests with --warn as an option at least

@rtfeldman Maybe you've actually been trying it with Elm 0.15, so hopefully I'm wrong about this, but it seems like 0.6.0 will be broken for Elm 0.15:

deadfoxygrandpa/elm-test (lower case) is only published for Elm 0.16-alpha, so Elm 0.15 (currently) requires deadfoxygrandpa/Elm-Test (Title Case). Also laszlopandy/elm-console/1.0.1 (the highest version that works with Elm 0.15 also has a different API than laszlopandy/elm-console/1.0.2, and elm-test might not work with the 0.15 version.

If 0.6.0 works for existing projects, then elm-test init is probably broken for Elm 0.15.

When a Debug.crash is hit the output looks like

$ elm-test
Success! Compiled 1 module.                                         
Successfully generated /var/folders/1x/1ftzf2l53wxf7bzbvsc9vtp40000gn/T/elm_test_11681-67517-t6bj0l.2vwj2u766r.js

which is a tad confusing. To replicate see https://github.com/opsb/elm-test-crash

Design

1. Have the CLI compile elmTestWorker.js and then take on the role of supervisor.
2. Supervisor spawns up to 4 node processes (~10mb overhead each), each running elmTestWorker.js
3. On startup, each worker distributes seeds and sorts the pending tests into a pending data structure. (Supervisor sends each worker a seed, either randomly generated or passed through from --seed CLI flag, so that they all independently arrive at identical pending data structures.)
4. The supervisor sends each one a test index to run, e.g. "worker 1: run test 1, worker 2: run test 2", etc.
5. Each worker finds the appropriate index in the pending data structure and runs it.
6. When a worker finishes, it reports the result back to the supervisor, and then waits for further instructions.
7. When the supervisor hears back from a worker, it adds that worker's result to a results map.
8. When the results map has contiguous unreported tests (e.g. tests 5-7 have all come back), it reports them to the end user (that is, prints them on the screen). Uncontiguous ones (e.g. 7 has come in, but still waiting on 5 and 6) means to hold off on reporting. This will make it so the output order is the same every time, regardless of what's going on in parallel.
9. Once a worker's response has been processed, the supervisor increments the current test index and sends it to that worker.
10. If a worker gets a text index that's out of bounds, that means there are no more tests to run, so the worker process stops waiting for messages and exits.
11. Once all workers have exited, the supervisor is done and can print a summary.

Since only the workers need to interact with Elm code, the CLI/supervisor can be written in any language.

Supervisor/Worker Protocol

Worker

• Receive configuration options (including the .js file to run) in argv
• Listen on stdin for integers corresponding to test indices
• Echo JSON test results after running each test
• When the received index is out of bounds, we're done; exit 0
• If any errors happen, exit 1

Supervisor

• Accept a --path-to-node CLI flag in case you don't want to use the version of node that's on your PATH for the workers. This would also enable testing out custom worker variants; instead of pointing it at a vanilla node binary, point it at something that does something fancier with the extremely basic communication protocol. (Default value: the node on your PATH)
• Detect # of cores available; spawn 1 node process per core
• Follow the steps as described above

Other Thoughts

https://github.com/rtfeldman/elm-web-workers doesn't seem necessary on Node, although we might want to use it to parallelize the tests in the browser.

I . got this json but 4 of the modules are on the wrong versions

{
    "version": "1.0.0",
    "summary": "Sample Elm Test",
    "repository": "https://github.com/user/project.git",
    "license": "BSD-3-Clause",
    "source-directories": [
        ".",
        "../src"
    ],
    "exposed-modules": [],
    "dependencies": {
        "elm-community/json-extra": "1.0.0 <= v < 2.0.0",
        "elm-lang/html": "1.0.0 <= v < 2.0.0",
        "mgold/elm-random-pcg": "3.0.0 <= v < 4.0.0",
        "elm-lang/core": "5.0.0 <= v < 6.0.0",
        "elm-community/elm-test": "3.0.0 <= v < 4.0.0",
        "rtfeldman/node-test-runner": "3.0.0 <= v < 4.0.0"
    },
    "elm-version": "0.18.0 <= v < 0.19.0"
}

--> Your elm-package.json has the following dependency:
    
        "elm-community/json-extra": "1.0.0 <= v < 2.0.0"
    
    But none of the versions in that range work with Elm 0.18.0. I recommend
    removing that dependency by hand and adding it back with:
    
        elm-package install elm-community/json-extra 2.0.0

--> Your elm-package.json has the following dependency:
    
        "mgold/elm-random-pcg": "3.0.0 <= v < 4.0.0"
    
    But none of the versions in that range work with Elm 0.18.0. I recommend
    removing that dependency by hand and adding it back with:
    
        elm-package install mgold/elm-random-pcg 4.0.2

--> Your elm-package.json has the following dependency:
    
        "elm-lang/html": "1.0.0 <= v < 2.0.0"
    
    But none of the versions in that range work with Elm 0.18.0. I recommend
    removing that dependency by hand and adding it back with:
    
        elm-package install elm-lang/html 2.0.0


Compilation failed for Main.elm

What if you could run elm-test --interactive and when it got to a test failure, it could ask you if you'd like to jump into a REPL session right there to debug?

The time-saving feature would be that everything inside the test's anonymous function would be in scope (plus the file's toplevel scope as usual), so you could play around with maximum ease.

I don't know how often that would actually come in handy, and it sounds like it'd take a lot of work, but it's an idea!

Maybe https://github.com/shamansir/node-elm-repl would be useful?

I've installed both elm and elm-test locally through NPM, however elm-test doesn't seem to be able to locate elm-make by default:

$ elm-test
Could not find Elm compiler "elm-make". Is it installed?

$ which elm-make
node_modules/.bin/elm-make

$ elm-make
Success! Compiled 0 modules.

$ elm-test --compiler $(realpath node_modules/.bin/elm-make)
Success! Compiled 81 modules.
Successfully generated /var/folders/g1/xqkylmwn4_x8mkyh5rf2cgkh0000gn/T/elm_test_1161013-51250-ldbbs7.gg24mo0f6r.js

elm-test
--------

Running 3 tests. To reproduce these results, run: elm-test --seed 738405972

As you can see, manually specifying the path does works, but could it be possible to make elm-test also look into node_modules/.bin by default?

elm-package.json contains the deps

"Apanatshka/elm-signal-extra": "5.7.0 <= v <= 5.7.0",
"circuithub/elm-json-extra": "2.2.0 <= v <= 2.2.0",
"circuithub/elm-list-extra": "3.7.1 <= v <= 3.7.1",
"circuithub/elm-maybe-extra": "1.5.1 <= v <= 1.5.1",
"circuithub/elm-result-extra": "1.3.1 <= v <= 1.3.1",
"elm-lang/core": "3.0.0 <= v <= 3.0.0",
"evancz/elm-effects": "2.0.1 <= v <= 2.0.1",
"evancz/elm-html": "4.0.2 <= v <= 4.0.2",
"evancz/elm-http": "3.0.0 <= v <= 3.0.0",
"evancz/elm-markdown": "2.0.0 <= v < 3.0.0",
"evancz/start-app": "2.0.2 <= v <= 2.0.2",
"NoRedInk/elm-rails": "3.0.0 <= v <= 3.0.0",
"NoRedInk/elm-string-extra": "1.1.1 <= v <= 1.1.1",
"rluiten/mailcheck": "3.0.2 <= v <= 3.0.2",
"rtfeldman/elm-validate": "1.1.1 <= v <= 1.1.1",

we should make sure that test/elm-package.json contains at least the packages

"Apanatshka/elm-signal-extra": "5.7.0 <= v <= 5.7.0",
"circuithub/elm-json-extra": "2.2.0 <= v <= 2.2.0",
"circuithub/elm-list-extra": "3.7.1 <= v <= 3.7.1",
"circuithub/elm-maybe-extra": "1.5.1 <= v <= 1.5.1",
"circuithub/elm-result-extra": "1.3.1 <= v <= 1.3.1",
"elm-lang/core": "3.0.0 <= v <= 3.0.0",
"evancz/elm-effects": "2.0.1 <= v <= 2.0.1",
"evancz/elm-html": "4.0.2 <= v <= 4.0.2",
"evancz/elm-http": "3.0.0 <= v <= 3.0.0",
"evancz/elm-markdown": "2.0.0 <= v < 3.0.0",
"evancz/start-app": "2.0.2 <= v <= 2.0.2",
"NoRedInk/elm-rails": "3.0.0 <= v <= 3.0.0",
"NoRedInk/elm-string-extra": "1.1.1 <= v <= 1.1.1",
"rluiten/mailcheck": "3.0.2 <= v <= 3.0.2",
"rtfeldman/elm-validate": "1.1.1 <= v <= 1.1.1",

Currently it seems that in order to run tests we need to have all the parent projects deps also listed in the test project's elm-package file. This is going to be hard to maintain. It would be good if we could have a mechanism to keep these in sync.

How about the first thing that the elm-test executable is to read the parent project's elm-package and write any missing deps into the test projects deps? It could also sync versions.

Currently, the code runs in the current directory unless the test file is in a tests/ directory. In that case, it jumps into the tests/ directory and runs the elm compiler there.

Rather than assuming a particular file structure, the code should search for elm-package.json to determine where the compilation root is.

To recreate, use a project that has many dependencies listed in tests/elm-package.json:

rm -Rf tests/elm-stuff
elm-test --watch

On the first run, elm-test will download the dependencies, but when the dependencies are downloaded, watch mode will see a bunch of new elm files appear under elm-stuff, and will kick off a bunch of simultaneous new builds while the first build is still downloading dependencies (and all the new builds will start downloading dependencies too!)

I think there are two things that ideally should both be done:

• Don't start a new build while a previous one is still running
• Don't watch files under elm-stuff (though maybe some people would actually want this?)

this and this fail in ci.js, but work if you run the same steps manually.

The proximal cause is that for some reason in ci.js, elm-test init generates a template file that is consistently missing an entire line of code. If I run elm-test init in my terminal, the line of code is not missing.

I'm getting a runtime error. 😞 To reproduce:

cd $TMPDIR
git clone [email protected]:mgold/elm-nonempty-list.git
cd elm-nonempty-list
elm-make

Result:

Starting downloads...

  ● elm-lang/lazy 1.0.0
  ● elm-community/elm-test 2.0.1
  ● mgold/elm-random-pcg 3.0.1
  ● elm-lang/core 4.0.5
  ● elm-community/elm-lazy-list 1.3.0
  ● elm-lang/html 1.1.0
  ● rtfeldman/node-test-runner 1.0.0
  ● elm-community/shrink 1.1.0
  ● elm-lang/virtual-dom 1.1.1

Packages configured successfully!
Success! Compiled 49 modules.                                       
Successfully generated /var/folders/ll/yf9q1nj56sb97s7twbtvl88c0000gn/T/elm_test_11685-7708-2zs4pk.yslkk5ipb9.js
undefined:1929
            throw new Error(
            ^

Error: You are giving module `Main` an argument in JavaScript.
This module does not take arguments though! You probably need to change the
initialization code to something like `Elm.Main.fullscreen()`
    at init (eval at <anonymous> (/usr/local/lib/node_modules/elm-test/bin/elm-test:86:37), <anonymous>:1929:10)
    at Object.eval [as callback] (eval at <anonymous> (/usr/local/lib/node_modules/elm-test/bin/elm-test:86:37), <anonymous>:1973:17)
    at step (eval at <anonymous> (/usr/local/lib/node_modules/elm-test/bin/elm-test:86:37), <anonymous>:2613:39)
    at Timeout.work [as _onTimeout] (eval at <anonymous> (/usr/local/lib/node_modules/elm-test/bin/elm-test:86:37), <anonymous>:2671:15)
    at tryOnTimeout (timers.js:232:11)
    at Timer.listOnTimeout (timers.js:202:5)

What's going on?

Hey,

It would be really nice to have an example fuzz test in the test file that is produced by init. I think it might encourage people to try out fuzz testing, or at least provide a quick code example for those of us too lazy to go and look up the docs!!

Thanks

If tests are taking too long to run, there's currently no way to tell which individual tests are slow.

A --verbose flag could let you see:

• Both successes and failures
• Durations for each individual test

The former is a prerequisite for the latter; if you only see durations of failures, that doesn't help much!

Currently gives: Object has no method 'execSync'

The default format of the test output is great, but it would be awesome to have something like JSON or TAP output so that the community can create custom formatters (like mocha's dot or spec outputs).

I wanted to open the conversation to see what some ideas could be.

It would be nice to be able to run a test run where each fuzz test did a much larger number of tests than the normal without having to change code.

that is being able to do this

elm test --fuzz-itterations 10000

Or via an environment variable, but probably better on the command line

I would like node-elm-test to support a --warn flag, which it would pass on to node-elm-compiler. It would help to improve the quality of the test code.

I would also suggest using a library like minimist or yargs for parsing the command line params in order to make the interface more flexible for the users and new flags easier to implement.

Hello feldman!

Must first say that I love your work.
Im wondering if there will be a 0.17 port of this related to the community-elm/elm-test?

See #29: the user had nodejs-0.10.25, which gives the following error:

$ elm-test init
elm-package.json already exists

/usr/local/lib/node_modules/elm-test/bin/elm-test:41
childProcess.execSync(elm["elm-package"] + " " + command + elmOptions, {
^
TypeError: Object # has no method 'execSync'
at execElmPackageSync (/usr/local/lib/node_modules/elm-test/bin/elm-test:41:20)
at Object. (/usr/local/lib/node_modules/elm-test/bin/elm-test:45:3)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Function.Module.runMain (module.js:497:10)
at startup (node.js:119:16)
at node.js:902:3

We should detect an old version of nodejs and give an error suggesting an upgrade. (0.11+ is required for execSync).

We've added Test.todo with the intent of the runners filtering out such tests, unless they are the only failing tests. Well, now that Test.Runner.isTodo exists it's time to implement that. The only trouble is that these changes are on master and not yet released, but we want the runners to be ready to go when we do release them.

I think it would be super helpful, if Configuration section of REAMDE.md mentioned all available configuration options.

• yes
• warn
• version
• help
• watch
• compiler
• seed
• report