nodejs / citgm Goto Github PK
View Code? Open in Web Editor NEWCanary in the Gold Mine
Home Page: https://www.npmjs.com/package/citgm
License: Other
Canary in the Gold Mine
Home Page: https://www.npmjs.com/package/citgm
License: Other
One additional check we may need to perform is a quick check against engines
in the package.json to make sure that the version of the module being tested will be compatible with the version of node used. Leaving this here as a remind to get that added in.
citgm-dockerify
is great for testing on linux, not so great for automating against other platforms.
It would be ideal to have the ability to automate launching VM images, pushing a node install, then running a (or multiple) citgm test on the launched image. Using something like https://www.npmjs.com/package/pkgcloud and cloud-init ought to do the trick. Just need to play around with it to get it working.
patched in pugjs/pug#2238 (comment)
waiting for it to be released
There are currently a number of modules in lookup.json that are flakey for a number of reasons.
Does not run nicely in child process
Needs color output on stdout
published version broken, fixed on master
requires special setup or env
What would we like to do with these modules?
We can easily implement a "flakey" check for modules that we know are broken but can be fixed
Should we do research on getting modules that require color output or do not run nicely in child processes?
Should we drop support for modules that require specific environments or setup outside of npm install?
One of the largest bottlenecks right now is downloading tar balls and running npm install
.
If we could smartly cache we could likely get away with only calling npm rebuild
This could likely be accomplished naively by not cleaning up the temp folders, but there might be a more elegant way.
Currently all modules that have a problem have been put in the blanket category of flaky allowing it to fail on all versions and platforms. It would be nice to be able to be more granular about flakiness.
Reasons a module may be flaky (thanks @mhdawson for coming up with this list)
Do people have ideas for other potential cases of flakiness?
I've set up a Jenkins job to start playing around with this: https://jenkins-iojs.nodesource.com/job/smoker, currently only hooked up to a fast Linux slave while I test this out.
Successful run here: https://jenkins-iojs.nodesource.com/job/smoker/6/nodes=debian8-64/console (well, "successful" in the "it runs" sense, I'm only running 3 modules and hapi is failing 4 tests).
Jenkins config has the following for Linux after cloning the repo you give it (default nodejs/node) at the commit provided:
First compile and install Node:
./configure --prefix=$(pwd)/smoker
make -j $(getconf _NPROCESSORS_ONLN)
make install
then set up citgm:
export PATH=$(pwd)/smoker/bin:$PATH
cd smoker
node -v
npm -v
npm install --prefix=$(pwd) --global --loglevel=error citgm
then tests
export PATH=$(pwd)/smoker/bin:$PATH
export npm_config_nodedir=$(pwd)
cd smoker
citgm --nodedir=$npm_config_nodedir --lookup --verbose through2@latest
citgm --nodedir=$npm_config_nodedir --lookup --verbose bignum@latest
citgm --nodedir=$npm_config_nodedir --lookup --verbose hapi@latest
Some notes:
--verbose
, is this intended? Should we have a less verbose option?--nodedir
doesn't pass down properly if the module isn't set up quite right, e.g. https://github.com/justmoon/node-bignum/blob/5f668a1c44324fd2ec074bafb1bd215883d8e856/package.json#L37 (although I can fix that one by removing that line, but it's a useful case), hence the use of export npm_config_nodedir=$(pwd)
to achieve the same thing.Is this roughly how you envisioned it being used @jasnell, perhaps you can give some feedback?
Currently we are not testing this... probably a good idea to do so
Originally mentioned in #5
Currently if tests fail we only get the message The canary has died
it would be nice to have more info without having to rerun the suite. Not 100% this is easily doable but we can use this issue to track progress
Across different systems and versions we are having inconsistent results testing request
on centos 5 / 6
verbose: npm-test: | not ok test-localAddress.js ......... 5/5
on ubuntu 1404
verbose: | Assert: "Test file timed out:
verbose: | test-localAddress.js",
verbose: | found: true
verbose: | wanted: false
verbose: | test-localAddress.js:0
On OSX 10.10 there seems to be some sort of rate at which it randomly fails during the phantomJS stages
many modules were failing in centOS
https://ci.nodejs.org/job/thealphanerd-smoker/14/
Turning off support for now, to be investigated
In some node-modules like request,
The npm test runs different tests and we only want to run test-ci .
How will we do it via citgm?
Thanks
"scripts": {
"test": "npm run lint && npm run test-ci && npm run test-browser",
"test-ci": "taper tests/test-.js",
"test-cov": "istanbul cover tape tests/test-.js",
"test-browser": "node tests/browser/start.js",
"lint": "eslint lib/ *.js tests/ && echo Lint passed."
},
PR sent to fix this
Looks like v13.0.0 was published 2 days ago. It was tagged with a proceeding v. Past released have had no proceeding character.
Rather than updating the lookup I have modified citgm-all to consider modules the don't fetch from npm as flaky, so our CI runs will at least pass.
I've followed up with the person who did the release and I'm going to wait to see what happens before I make changes to the lookup table.
[nodetest@test vmTest]$ citgm -lv request@latest
not ok 4 should be equivalent
operator: deepEqual
expected: [ 'https connect to localhost:6767', 'http response', '200 ht tp ok' ]
actual: [ 'err tunneling socket could not be established, cause=certifi cate has expired' ]
at: Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
...
not ok 5 should be equivalent
operator: deepEqual
expected: [ 'https proxy to http', 'http response', '200 http ok' ]
actual: [ 'err certificate has expired' ]
at: Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
...
not ok 6 should be equivalent
operator: deepEqual
expected: [ 'https proxy to http', 'http response', '200 http ok' ]
actual: [ 'err certificate has expired' ]
at: Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
...
not ok 7 should be equivalent
operator: deepEqual
expected: [ 'http connect to localhost:16167', 'https response', '200 h ttps ok' ]
actual: [ 'http connect to localhost:16167', 'err certificate has expir ed' ]
at: Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
...
/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/node_modules/tape/index.js:75
throw err
^
Error: certificate has expired
at Error (native)
at TLSSocket. (_tls_wrap.js:929:36)
at TLSSocket.emit (events.js:104:17)
at TLSSocket._finishInit (_tls_wrap.js:460:8)
Assert: "should be equivalent",
found: [ 'err tunneling socket could not be established, cause=certific ate has expired' ]
wanted: [ 'https connect to localhost:6767', 'http response', '200 http ok' ]
Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
Assert: "should be equivalent",
found: [ 'err certificate has expired' ]
wanted: [ 'https proxy to http', 'http response', '200 http ok' ]
Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
Assert: "should be equivalent",
found: [ 'err certificate has expired' ]
wanted: [ 'https proxy to http', 'http response', '200 http ok' ]
Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
Assert: "should be equivalent",
found: [ 'http connect to localhost:16167', 'err certificate has expire d' ]
wanted: [ 'http connect to localhost:16167', 'https response', '200 http s ok' ]
Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
ok test-unix.js ..................... 3/3
total ......................... 1326/1331
not ok
npm ERR! Linux 2.6.32-431.17.1.el6.s390x
npm ERR! argv "/usr/nodetest/T32_12/bin/node" "/usr/nodetest/T32_12/bin/ npm" "run" "test-ci"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! code ELIFECYCLE
npm ERR! [email protected] test-ci: taper tests/test-*.js
npm ERR! Exit status 5
npm ERR!
npm ERR! Failed at the [email protected] test-ci script 'taper tests/test-.js'.
npm ERR! This is most likely a problem with the request package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! taper tests/test-.js
npm ERR! You can get their info via:
npm ERR! npm owner ls request
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/npm-debug.log
npm ERR! Test failed. See above for more details.
Error |failure |The canary is dead.
Info |rm.tempdir |/tmp/9eab4220-cc8f-43ca-bb1b-71
Info |done |https://github.com/request/requ
| |.0.tar.gz
Currently we have a single large lookup table.
If we split the look up into subsystems (as we do our commits) we could use a flag to specify only testing a subset of the packages.
I'm considering the possibility of adding a support for a npm smoketest
convention. Essentially, if module developers want to include a smoketest
script in their package.json as an alternative to npm test
, then citgm
would choose to run that as an alternative. Doing so would allow module developers to create subsets of the tests that make the most sense for smoke testing, or perform additional set up that may be necessary independently of the normal test suite.
A question was raised about the possibility of having citgm recursively check the dependencies of a module as well as the module itself. We are grabbing the package.json so doing so would be possible but it could potentially be a significant grind to get through everything in one go. Not convinced it's something we should do but wanted to solicit opinions.
@nodejs/smoke-test .. what do you think?
Currently it does not look like we are going to be using citgm-dockerify in production.
The binary does not currently have tests.
I'd like to suggest deprecating the binary for now so we can focus on core requirements
/cc @jasnell
-k|--hmac
option is working correctlyWe should see a 20% speed up in npm install using this change.
npm is not resolving all the dependencies causing the module to fail
Master is working. This issue can be closed when we revert eslint installing from master
the current error message is
error: failure | No project downloaded
It would be nice to have more information, like perhaps that the package doesn't exist on npm, or the link provided returned a 404, etc...
v5 and up throw an error.
throw new Error('Unsupported Node version: ' + nodeVersion);
seems to be coming from mock-fs
currently the examples in the readme include -v
but do not include a loglevel, and as such they fail 😢
1) CLIEngine executeOnText() should report one
verbose: | message when using local cwd .eslintrc:
verbose: | TypeError: Path must be a string. Received
Looks related to #80
This might be useful when chasing down really odd bugs on specific systems.
Chalk is failing within citgm. I think the styling is currently being stripped.
Not sure exactly what is happening, but we can use this issue to track failures due to npm install hanging.
The tool needs to be integrated into the CI environment so that fully automated runs are possible. The current design of the tool does not make that easy. /cc @thealphanerd
Modules may become flaky / unflaky at a very fast rate. Would people be open to changes to module flakyness to be merged without code review? (As long as it is documented)
It is fine on v5
176 passing (494ms)
1 failing
1) bodyParser.urlencoded() with parameterLimit option with extended: false should work with Infinity limit:
AssertionError: 0 == 10000
+ expected - actual
-0
+10000
at test/urlencoded.js:666:12
at Test._assertFunction (node_modules/supertest/lib/test.js:247:11)
at Test.assert (node_modules/supertest/lib/test.js:148:18)
at Server.assert (node_modules/supertest/lib/test.js:127:12)
at emitCloseNT (net.js:1525:8)
verbose: npm-test: | 1 failing
verbose: npm-test: | 1) sourcemap basic inline:
verbose: | TypeError: test/sourcemap/basic.inline.styl:10:17
verbose: | 6| width: 500px
verbose: | 7| margin: 0 auto
verbose: | 8|
verbose: | 9| h2
verbose: | 10| color: green
verbose: | -----------------------^
Will dig in more soon
it may be easier to add known failures to the arguments rather than constantly modifying the lookup
perhaps @jdalton can chime in
verbose: |
verbose: | Error: Cannot find module '../lib/fp/convert.js'
verbose: | at Function.Module._resolveFilename
verbose: | (module.js:325:15)
verbose: | at Function.Module._load (module.js:276:25)
verbose: | at Module.require (module.js:353:17)
verbose: | at require (internal/module.js:12:17)
verbose: | at Object.<anonymous>
verbose: | (/private/var/folders/ty/q7q6b07j5r3c7nvnpzm6hkq4…
verbose: | 0000gn/T/1287562e-5561-4d58-8c21-eb82a940002f/lod…
verbose: | ash/test/test-fp.js:47:28)
verbose: | at Object.<anonymous>
verbose: | (/private/var/folders/ty/q7q6b07j5r3c7nvnpzm6hkq4…
verbose: | 0000gn/T/1287562e-5561-4d58-8c21-eb82a940002f/lod…
verbose: | ash/test/test-fp.js:931:3)
verbose: | at Module._compile (module.js:397:26)
verbose: | at Object.Module._extensions..js
verbose: | (module.js:404:10)
verbose: | at Module.load (module.js:343:32)
verbose: | at Function.Module._load (module.js:300:12)
Turns out that process.stdout in a child_process is created using net, whereas in node proper it is made using tty. If a call to any of the methods tty exposes that net does not have, everything will explode
I have submitted a PR to fix this nodejs/node#5061
1 failing
1) socket.io socket should handle very large binary data:
Error: timeout of 2000ms exceeded
When I did the io.js 2.5.0 release, I used CITGM to do some smoke testing. The experience was generally positive. Some complaints / things that would be nice to have:
X out of Y tests passed
. It's also not obvious what breaks because of the platform and what is just broken in the module. I treated it as "oh there are a few failures, but the module still works."React has been moved to flakey due to this. The engine field has been bumped in master. Once a new version is cut react can be moved from flakey and this issue closed
The entire test suite for SPDY is broken on master
verbose: | 1) SPDY Client regular h2 ssl mode should send GET
verbose: | request:
verbose: | Error: timeout of 2000ms exceeded. Ensure the
verbose: | done() callback is being called in this test.
verbose: npm-test: | 2) SPDY Client regular h2 ssl mode "after each"
verbose: | hook:
verbose: | Uncaught Error: socket hang up
verbose: | at TLSSocket.onclose
verbose: | (node_modules/spdy-transport/lib/spdy-transport/c…
verbose: | onnection.js:186:15)
verbose: | at TCP._onclose (net.js:471:12)
verbose: |
verbose: | 3) SPDY Client regular h2 plain mode should send
verbose: | GET request:
verbose: | TypeError: Cannot read property 'request' of
verbose: | undefined
verbose: | at Context.<anonymous> (test/client-test.js:77:26)
verbose: |
verbose: | 4) SPDY Client regular h2 plain mode should send
verbose: | POST request:
verbose: | TypeError: Cannot read property 'request' of
verbose: | undefined
verbose: | at Context.<anonymous> (test/client-test.js:95:26)
verbose: |
verbose: | 5) SPDY Client regular h2 ssl mode "before each"
verbose: | hook for "should receive PUSH_PROMISE":
verbose: | TypeError: Cannot read property 'call' of
verbose: | undefined
verbose: |
verbose: |
verbose: | 6) SPDY Client regular h2 plain mode should
verbose: | receive PUSH_PROMISE:
verbose: | TypeError: Cannot read property 'request' of
verbose: | undefined
verbose: | at Context.<anonymous>
verbose: | (test/client-test.js:109:26)
verbose: |
verbose: | 7) Uncaught error outside test suite:
verbose: | Uncaught TypeError: Cannot read property
verbose: | 'currentRetry' of undefined
The CITGM testing shows a PPC specific failure for the npm tests on eslint. This is not a problem running the module (ie eslint itself does not have a problem) but because of dependencies in the test.
https://ci.nodejs.org/job/thealphanerd-smoker/21/nodes=ppcle-ubuntu1404/console
Myles and I have identified that the issue is a missing binary in the phantomjs module so we'll see what we can do to have that added. Looks like phantomjs chooses its platforms based at least partially on what Node needs so hopefully they will be receptive to adding the PPC binaries now that Node builds for those platoforms.
I'm currently using citgm-all with a lookup.json file as part of Jenkins CI, and it'd be useful to have citgm-all output a TAP file in the same way node tools/test.py does with its --progress=tap
option. That way, with the Jenkins TAP plugin it'd be really easy to redirect the citgm-all output to a .tap file for a range of tests ( avoiding having to dig through the console log on every failure).
The individual npm modules often have their own TAP outputs for their tests, but an overall TAP file with one TAP result for each module would be useful.
I made a draft of what I imagined the tap output would look like, let me know if it seems feasible. I'm not sure how much work it would take to add this.
1..2
ok 1 express
---
duration_ms: 100.137
...
not ok 2 lodash
# not ok 20 lodash-test-connection
# ECONNRESET
#
---
duration_ms: 60.88
...
Feedback/criticism welcome!
one of the current shortcomings of the tool is that it takes whatever the modules npm tests generates and just pipes those to stdout, which means in order for us to know if there are any failures, we have to go digging. Which is time consuming. The tool needs to be refactored to generate consistent usable output. /cc @thealphanerd
Some node-modules only support 64 bit node.js and not 32 bit node.js
Some do not support certain platforms, is there a way to skip running tests based on different conditions?
arch, os, platforms,.. node.js bits.?
Moving discussion from https://github.com/rvagg/iojs-smoke-tests/issues/1
I'm +1 on adding Hapi given its heavy test burden and enterprise popularity. Does it need anything more than an entry on the README?
Would be nice to have a flag that would print formatted output ready to be pasted in a github issue
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.