https://www.npmjs.com/package/jest-light-runner

Hi, thanks for this awesome package; it has sped up my tests 10x, so I'm very happy with it!!

When I was first considering adopting jest-light-runner, though, one of the things that I most wanted to understand was exactly what type of test isolation I'd be giving up in exchange for the speed boost. I wasn't able to find much documentation on exactly how jest's test isolation works, so I mostly relied on the jest-light-runner README. After playing around with jest and this runner, though, I think this note in the readme is a bit incomplete:

Tests isolation. Jest runs every test file in its own global environment, meaning that modification to built-ins done in one test file don't affect other test files. This is not supported, but you can use the Node.js option --frozen-intrinsics to prevent such modifications.

In particular, from my experiments (please correct me if I'm wrong), it seems like Jest not only evaluates each test file it its own node VM, but also gives each of those VMs a distinct ESM cache (to use when linking modules into the VM), so that mutable ESM exports are also isolated (and re-evaluated) between test files. If that is correct, then it seems important to note that in the README, because --frozen-intrinsics obviously won't flag/prevent mutating module exports.

(As an aside, it might also be worth documenting that --frozen-intrinsics doesn't work with ts-node, I discovered, because the TS compiler sets some properties on the global Error object.)

Anyway, like I said, I think this library is awesome! I'm just opening this issue to double check that my understanding is correct, and, if so, note the confusing bits that I ran into when adopting this library (which did break a few of my tests at first), so that hopefully they can be clarified for future users.

Seems the "serializer" is added globally, maybe because we are not running test files isolatedly?

I haven't dug into the problem, I'll post what I found.

Hi there, I'm trying to use the fake timers from jest, but I can't get them to work. I'm using jest-light-runner with esmock, and my dates are handled by dayjs.

Non-working snippet:

const now = dayjs();

beforeAll(async () => {
  await db.raw(`ATTACH DATABASE ":memory:" AS ${Customer.schema}`);
  await db.migrate.latest();
  jest.useFakeTimers();
  jest.setSystemTime(now);
});

Resulting in:

          OptIn {
    -       "effectiveAt": "2023-06-30T08:47:48.437Z",
    +       "effectiveAt": "2023-06-30T08:47:48.463Z",
            "name": "text",
            "value": true,
          },

Working snippet:

const now = dayjs();

beforeAll(async () => {
  await db.raw(`ATTACH DATABASE ":memory:" AS ${Customer.schema}`);
  await db.migrate.latest();
  MockDate.set(now);
});

Am I doing something wrong? I also tried legacy timers in jest to no avail.

I'm guessing that this falls under "Unsupported Jest features", but I want to confirm:

Our project contains a mix of front-end (React) and generic (business logic) code, so we let Jest use its default node environment then configure individual test cases to use JSDom via @jest-environment jsdom docblocks. This is faster than initializing JSDom everywhere.

In a brief experiment with jest-light-runner, I saw that these tests are failing. I assume that, because jest-light-runner doesn't use vm to isolate tests, it doesn't support @jest-environment docblocks and must run every test in the same environment?

Thank you.

Currently all of the hooks registered for a test suite are run before the test itself is executed. The afterEach and afterAll hooks should delayed until the test is completed.

PR with a fix to follow! 👍

Using fake timers in multiple tests freezes the execution, with no other option but to kill the process.

import dayjs from "dayjs";

const now = dayjs();

beforeEach(async () => {
  jest.useFakeTimers().setSystemTime(now.toDate());
})

afterEach(async () => {
  jest.useRealTimers();
})

Calling jest.useRealTimers(); in afterEach as suggested here doesn't help unfortunately. Sorry I took so long go get back to creating a dedicated issue.

I've been trying to migrate a module to using this test runner (thank you so much!).
Unfortunately I got stuck behind an error: TypeError: jest.setTimeout is not a function
I am not sure if this is a problem in this repo to be honest, but suspect it might not be supported here? If so, is there an alternative?
Thanks!

The following test passes with jest but not jest-light-runner. The beforeAll behaviour appears to be correct, though.

describe('beforeEach execution order', () => {
  const calls = []
  beforeEach(() => {
    calls.push('outer')
  })

  describe('nested', () => {
    beforeEach(() => {
      calls.push('inner')
    })

    it('expect outer beforeEach before inner', () => {
      expect(calls).toEqual(['outer', 'inner'])
    })
  })
})

As suggested by Simen, it should be enough to read this.#config.maxWorkers in

I'm trying to use jest-light-runner on a testing project, and can't really understand the process I'm supposed to be following, if there is support for such a setup.

What I'm currently doing is adding the runner: "jest-light-runner" option to my jest config (which by itself will output errors for Unknown file extension ".ts").

I understand this is due to on-the-fly compilation not being supported, as mentioned by the docs. It is mentioned that I should use something like ts-node/esm, which I set up with the flag --loader ts-node/esm as part of NODE_OPTIONS.

Doing so is getting me random errors (from pino, our logger, not sure why). Is there anything else I should do? Does typescript support actually rely on this experimental node option, or is there anything else I'm missing?

Hi there, thanks for the great project!

I consider jest's partial test isolation to be more trouble than it's worth and would rather switch to jest-light-runner, don't mess around with globals (expect with jest.spyOn) and use the built-in jest features to revert these changes after every test.

However, it seems that the resetMocks and restoreMocks config option no longer works with jest-light-runner (probably a separate bug), so I tried working around that using a custom setup script (with setupFilesAfterEnv).

Actual behavior

But it seems that beforeAll, beforeEach, afterEach, afterAll simply do nothing when called from such a script, for example:

setup-after-env.js:

console.log("setup-after-env")
beforeAll(() => console.log("setup beforeAll"))
beforeEach(() => console.log("setup beforeEach"))
afterEach(() => console.log("setup afterEach"))
afterAll(() => console.log("setup afterAll"))

demo.test.js:

it("should work", () => console.log("test"))

With jest-light-runner, this only prints:

setup-after-env
test

edit: Here's a minimal reproduction repo

Expected behavior

setup-after-env
setup beforeAll
setup beforeEach
test
setup afterEach
setup afterAll

It's possible that this is already working, because Babel uses this runner and collects a coverage when running it's tests: https://github.com/babel/babel/runs/5789483528. If it already works great, I just need to document it.

It's also possible that it's only working partially, or that it's not working at all and that Babel's coverage reporting is based on something else.

(cc @psychobolt - jestjs/jest#9430 (comment))

It's really useless to show error like this

SyntaxError: Unexpected token '('
    at Loader.moduleStrategy (internal/modules/esm/translators.js:140:18)
    at async link (internal/modules/esm/module_job.js:42:21)
error Command failed with exit code 1.

I tried a little bit, but seems there is no way to get the full stack.

nodejs/node#49441

I'm doing image snapshot testing, and it's very slow. The exact same file (no .concurrent) takes ~21 seconds with this runner, and only ~11 seconds with Jest's built-in. It has 58 snapshots, each invoking sharp to rasterize an SVG to PNG and passing it to jest-image-snapshot. Could I be missing something in my setup? I'm using ts-node/esm on top, just in case (and ts-jest for Jest).

Please expand the coverage report section of the README to include minimal steps to set up coverage reporting. Such that people with lesser knowledge of babel and loaders can follow.

Included discussion was based on my personal experience. For context I use pnpm with workspaces via rush.

Discussed in #71 (comment)

So I'v finally got it working.

1. Install "babel-register-esm", save dev
2. In ./jest.config.mjs

   collectCoverage: true,
   coverageProvider: "babel",
   coverageReporters: ["text"],

3. Add file ./babel.config.json

   {
     "plugins": ["babel-plugin-istanbul"]
   }

4. Run tests with NODE_OPTIONS='--loader=babel-register-esm' jest
   

It helps tracking down hard-to-debug bugs.

(cc @jasonLaster)

I am experimenting with the tsx/@esbuild-kit/esm-loader together with jest-light-runner, and the inclusion of the current date when importing the test file results in an ERR_UNKNOWN_MODULE_FORMAT error (when --experimental-specifier-resolution=node is used, Node 18.16.1. Without this option I am running into ERR_MODULE_NOT_FOUND because of the .ts extension).

I came to realize this is because jest-light-runner loads test modules using an additional query parameter, which causes issues with the tsx loader. Note that the ts-node/esm loader does work, I presume they strip query parameters from the path before passing it through.

Removing the + "?" + Date.now() part avoids this problem, allowing the tests to run nicely (and fast, so fast! 🚀) in runInBand mode (there's an unrelated issue when using workers, but I haven't looked into that sufficiently yet to tell what's going on).

It is not entirely clear to me what the date is trying to achieve. Is it for circumventing Node's module cache during watch mode? Should tsx perhaps identify the query param and strip it (which I assume is what ts-node does)?

I'm not using require/import mock.

The following snippet works with mocking

// working
class Adder {
  constructor(public getValueToAdd: () => number) {}

  add(digit: number) {
    return this.getValueToAdd() + digit;
  }
}

test("testing mock", () => {
  const mockGetValueToAddFunction = jest.fn().mockReturnValue(1);
  const adder = new Adder(mockGetValueToAddFunction);

  const result = adder.add(2);

  expect(result).toBe(3);
  expect(mockGetValueToAddFunction).toHaveBeenCalled();
});

However adding

beforeEach(() => jest.clearAllMocks());

results in

Test suite failed to run

    TypeError: Cannot read properties of undefined (reading 'length')

      at file:/home/xxxx/node_modules/jest-light-runner/src/worker-runner.js:222:29
          at Array.filter (<anonymous>)
      at toTestResult (file:/home/xxxx/node_modules/jest-light-runner/src/worker-runner.js:222:8)
      at default (file:/home/xxxx/node_modules/jest-light-runner/src/worker-runner.js:78:10)
      at async /home/xxx/node_modules/piscina/dist/src/worker.js:141:26

Not sure how it works.

Feature request: would be awesome if it supported typescript compilation (no type-checking needed).

Maybe something simple like esbuild transformer can be added for *.ts files, that'll cover 99% of the common needs.

Awesome runner, by the way!

jest-light-runner is working with esmock here iambumblehead/esmock#84

This is awesome :) thank you

I noticed that in IntelliJ, all tests have a reported duration of "0ms". Digging deeper revealed that the test result sent to the reports contains duration: NaN.
The typedef of the test result does contain a duration: number field, but it's never set - see skip case and execution case.

I'll have time on monday to contribute a fix for this, using the same pattern as for the total test duration calculation.

In the babel repository, it takes 42 seconds to run all tests locally (12 workers), and 35 seconds for babel-cli alone.

Because of the large number of subprocess operations involved, serial is very slow.

Perhaps we can make testing faster by supporting concurrency, greatly improving our local development experience.

I'm willing to try to implement this feature (probably not soon) and open this issue for your opinion.😃

tinypool is a lightweight fork of piscina with useless features like NAPI removed, it might be a good fit for this module.

Jest has globally available aliases for test.skip (xit) and describe.skip (xdescribe). .
With jest-light-runner, using these causes the test to fail with:

    ReferenceError: xit is not defined

As an easy workaround, I used the non-aliased test.skip and describe.skip. I'm open to providing a PR for this issue

When tests fail in my typescript project then Jest shows JavaScript stacktraces which are not very helpful. This can be fixed by doing this in every test file:

import "source-map-support/register";

Maybe it would be a good idea to do this by default directly in jest-light-runner?

As per the title, support for a custom snapshot resolver would be a great addition. A follow up PR with the implementation will be on the way shortly! 👍

Thanks for this exciting project! I have been toying with the light runner a couple times now and it is hugely impressive, yielding up to two orders of magnitude performance improvements (going from ~400s to ~5s for ~2500 tests in ~400 suites, using --runInBand). Jest's isolation is hurting us a lot, even though we don't need it.

I am experiencing an issue where the start message that is sent over a test's MessageChannel may arrive after the tests' Promise has already resolved, i.e. the finished callback has already been executed. This results in excessive status message logging, as the late start event does activate the test as "running" even though it has already finished, rendering it in the "running" state forever (and consequently generating excessive status messages as more and more tests become considered "running").

I don't know exactly why/when this happens, nor do I know what the delivery guarantees are w.r.t. MessagePort message timing vs the micro task queue. My observations are with the InBandPiscina mode and this may be relevant for this to occur, considering that Piscina would otherwise communicate over message channels itself.

Potential solutions

I see two approaches that would fix my issue:

1. Detect when a start message is received after the promise has resolved, then ignoring the start message entirely.
2. Delaying the finish notification until after the start message has been received, then notifying the status runner of both events simultaneously (possibly with a microtick between the start and end notification, if needed)

I have tested locally with approach 1 and that seems to work just fine, but option 2 results in more consistently communicating test statuses without skipping status transitions (not sure if there's any invariants in this area).

I would be happy to contribute either fix, just opening this issue for discussion.

Custom matcher libraries like jest-extended suggest to load the extension by specifying setupFilesAfterEnv in the Jest configuration. This doesn't work with jest-light-runner, it is simply ignored so I have to load jest-extended in every test file manually. Maybe it would be possible to support the setupFilesAfterEnv configuration?

Instead of using Workers, use cluster module ir child_process. This way, they would be full y independent processes so they would not share global atte, the same way Jest default environment does, and Aldo native chdir can be used.

Hi, I'm getting the following error on subsequent test runs, (as in, the first run works fine, but any runs after making any changes give this error) when running with --runInBand and --watch:

    FAIL  src/__tests__/bar.js
  ● Test suite failed to run

    Your test suite must contain at least one test.

      at onResult (node_modules/@jest/core/build/TestScheduler.js:172:18)
          at async Promise.all (index 0)

I made a mini reproduction repo here with instructions in the readme:

https://github.com/cjroebuck/jest-light-runner-runInBand-watch-bug-repro

I have tracked down the issue to this line https://github.com/nicolo-ribaudo/jest-light-runner/blob/main/src/worker-runner.js#L99

It seems that when run with --runInBand, the dynamic import when using InBandPiscina class is cached on subsequent test runs, and jest-circus fails to pick up the tests.

A hacky fix I made for this is to stick a random hash on the end of the import file path as a query parameter to 'bust' the import module cache, i.e.:

await import(pathToFileURL(testFile));

becomes

await import(pathToFileURL(testFile)+"?"+Date.now());

This works, but feels quite hacky and would probably lead to memory leaks if used in watch mode for many hours.

Prettier 3 is out now, but it's incompatible with inline snapshots. The workaround seems to be to point prettierPath in jest.config.js to a copy of [email protected]. However, jest-light-runner seems to ignore that setting, and instead hardcode the prettier path.

Switching to jest-light-runner with tests that use import {it, expect} from "@jest/globals" currently doesn't work. It's easily fixed by just removing this import and using those magically injected globals, but this way ESLint reports undefined variables and I lose my IDE's type features.

So could you either inject your own globals into from "@jest/globals" imports or expose your own globals, so I can use import {it, expect} from "jest-light-runner"?

Snapshot for removed test()s not removed. I have to remove the .snap file and regenerate it.