bearror / oletus Goto Github PK

Hi @bearror !

I'm still a very active and happy user of the project, and as such have recently run into a subtle yet problematic issue.

How to reproduce

Run oletus as normal with a test/whatever.mjs file:

import test from 'oletus'
test ('test', () => new Promise (() => {}));

Expected result

Some error about the test timing out, or even just a hanging process.

Actual result

 0 passed  0 failed  in 0.046s

Background

This problem has bitten me in a sense that I thought I had all my use-cases covered and my code had no bugs. However, some tests were simply being ignored because I did have an issue in my code that made it so the Promise never settled.

I'm opening this issue to discuss a matter that doesn't have a straight-forward enough solution for a PR.

My code uses the .stack property in Error instances. Overriding Error.prepareStackTrace before running a test, as done in the snippet below, means that while I'm testing this functionality, my code behaves differently from how it should.

I have attempted to fix this issue without success, by moving the location where prepareStackTrace is overridden closer to where .stack is accessed by Oletus:

https://github.com/Avaq/oletus/blob/309e79c0e19b2a949aedc4d26aeb89fc8f19744d/oletus.mjs#L18-L21

Yet I realize that even if I were successful, there is another issue with Oletus' approach to formatting errors, and that is that it assumes that all assertions are made directly at the testing location, and thus that only the first stack trace entry is relevant to the user.

For me, this is not the case. I have abstracted over assertions, and therefore Oletus shows me the same location for every failing assertion. It might therefore be worth considering the removal of the custom error formatting logic altogether, and just presenting the user with the default e.stack as a message.

If you do just present e.stack, then a user can always modify the output from that by overriding Error.prepareStackTrace themselves.

Reported paths of failing tests are Incorrect: they start with '/'. This prevents IDE to jump to the file from an embedded terminal.

Hi!

I have often difficulties to locate which assertion fails in my unit tests. Let's just take an example to illustrate this.

Usually my unit tests have several assertions, such as:

import test from "oletus"
test("example", t => {
    t.ok(true)
    t.ok(false)
})

The reported location is a bit evasive:

$ oletus test.js
x error
test.js:2

We don't know whether it is the first or the second assertion that failed. It could be nice to get the line where the assertion failed:

$ oletus test.js
x error
test.js:4

However I don't see how to get this location in a reliable way. Indeed: we do not know which assertion library is used and which test pattern are used. The current approach (reporting the test location) seems more reliable.

A possible approach could be to report a reliable location (the location currently reported) and an unreliable location:

$ oletus test.js
x example
test.js:2
test.js:4

However, I am not convinced by this. Have you some thoughts?