Improve how Flaky modules are handled about citgm HOT 5 CLOSED

/cc @jasnell @rvagg @nodejs/testing

from citgm.

Also should note that Node/V8 have a fairly flexible way to exclude tests using their status files so may be a good pattern to follow. They allow an exclude base on the platform as well as a run but allow to pass or run be allow to timeout etc.

from citgm.

Clarification: broken for all versions / broken on specific version ... does that means version of the module or versions of node? For instance, one could imagine that a module that is stable on v4.x could become flaky on v5.x

from citgm.

I am imagining that we will continue to only test the latest version of published modules. Thus Broken for all / broken on specific would refer to node version as opposed to module version.
This may get tricky as specific modules may not support LTS.

from citgm.

In terms of only testing the latest we ran into a few issues with that:

1. If you always test the latest you don't know without investigation if the breakage is from a change in the module or a change in the runtime
2. The latest may not support older streams like 0.12.X

We ended up doing 2 sets of runs

1. Latest runtime, fixed set of npm module levels

2. Last shipped runtime level, latest modules

from citgm.