We need a scalable (in terms of API size) documentation. I am thinking now markdown files in this repo + gitbook generated website.

toThrow is pretty limited now - it supports only full matching for error strings.

It should support following cases (from code):

// @todo: overloads like:
// .toThrow() throws anything
// .toThrow(/message/)
// .toThrow(ErrorClass)
// .toThrow(ErrorClass, 'message')
// .toThrow(ErrorClass, /message/)
// SINCE toThrow() is reserved for catching anything we need to do a extra work like for example: toThrow(expect.AUTOFIX)

Related: #113 (comment)

Probably error messages should be moved from validators to matchers too. This way we will get with super simple way to create validators for existing matchers and matchers will be able to return richer errors in case of mismatch.

This issue facilitates discussions about matchers design

Problem of having multiple ways to match the same thing

toEqual is quite a beast and thus many matchers can be implemented just by using it:

1)
expect(true).toEqual(true)
// instead of
expect(true).toBeTrue()

// same goes for other primitive values like (false/null/undefined)

2)
expect("abc").toEqual(any(String))
//instead of
expect("abc").toBeInstanceOf(String)

I had this problem on my mind for some time already and I initially thought that avoiding this duplication entirely is a good thing but I believe that having some level of consistency with chai can beneficial. For example, I think 1) is fine (it's clear and shorter 🤔)

Any thoughts on this one?

Essentials matchers to implement in the first batch

• toBeTruthy
• toBeFalsy
• toThrow (match message and match error type as separate matches? to read: chaijs/chai#655)

Asymmetric:

• any (or anyInstanceOf? much longer, much more explicit... i need to admit that any was confusing for me when I first saw it)
• anyString / anyStringMatching / anyStringContaining
• anyNumber

Adding the code below to the smartEq test suite causes it to fail with the stack trace:

      ...
      at C:\Users\sz-piotr\earl\packages\earljs\src\validators\smartEq.ts:78:16
      at Array.map (<anonymous>)
      at smartEq (src\validators\smartEq.ts:73:32)
      at C:\Users\sz-piotr\earl\packages\earljs\src\validators\smartEq.ts:78:16
      at Array.map (<anonymous>)
      at smartEq (src\validators\smartEq.ts:73:32)
      ...

it('compares recursive objects', () => {
  interface Node {
    prev?: Node,
    next?: Node,
    value: any
  }
  const node = (value: any): Node => ({ value })
  const list = (...values: any[]) => values.map(node).map((node, index, nodes) => {
    node.prev = nodes[index - 1]
    node.next = nodes[index + 1]
    return node
  })[0]

  const a = list(1, 2, 3)
  const b = list(1, 2, 3)
  const c = list(4, 5)

  expect(smartEq(a, b)).to.be.deep.eq({ result: 'success' })
  expect(smartEq(a, c)).to.be.deep.eq({ result: 'error', reason: 'value mismatch' })
})

One of Jest's weaknesses is that it doesn't use assertion functions based on the matchers.

See:

DefinitelyTyped/DefinitelyTyped#41179

The website: https://earljs.dev/
The package: earljs
The repo: earl
The organisation earl-js

Can we change everything to be earljs?

expect.arrayOfLength(n) and validator .toBeArrayOfLength(n)

Currently it's

toHaveBeenCalledExactlyWith(this: Expectation<Mock<any[], any>>, expectedCalls: MockArgs<T>): void;

and should be

toHaveBeenCalledExactlyWith(this: Expectation<Mock<any[], any>>, expectedCalls: MockArgs<T>[]): void;

to account for asserting all of the calls that were performed

It'd be cool to have a matcher for a subset of object properties. Eg.:

expect({a: 1, b: 2, c: 3}).toContainSubset({a: 1, c: 3})

Listing features is cool and all, but code says more

Tired

(Jest's website is still a tad better, because they show CLI output when you scroll down)

Wired

GraphQL Code Generator docs are powered by docusaurus too, so it's not that we can't do anything fancy.

Rename stringContaining to stringMatching and make it support:

stringMatching(string) - acts like string containg
stringMatching(/regexp/) - uses regex to do matching

Not everyone uses snapshots but the current implementation comes with many dependencies so it makes sense to extract it to a separate plugin.

@sz-piotr you're gonna love it :->

Possible names for a shallow eq validator:

• toShallowEqual
• toBe (jest style)

Personally I like toShallowEqual - it also goes well with toLooseEqual that we already have.

earl can support snapshot testing.

Reasoning

Snapshot testing is pretty popular in Jest. My personal opinion is that it's a cool feature that is very frequently overused and misused. Nevertheless, sometimes it's quite useful.

Implementation

By integrating with a test runner we can recognize currently running test case and automatically name snapshots, so we will end up more or less with similar API as Jest. Furthermore, I am sure that we can use jests snapshots serializers.

TODO:

• upgrade docozaurus
• add searchbox
• add example to the main page: Releated: #142
• add somewhere link to the changelog
• update CONTRIBUTING.md
• add playground

earl should support plugins to extend available validators and extractors to specific cases.

Autoloading

Packages matching naming schema: earljs-plugin-x should be autoloaded by earl during test initialization (requires test runner integration).

Type safety

Plugins should use declaration merging to extend earljs types. It's not the nicest but I don't see any other way.

Plugin API

Plugin is a function returning PluginConfig.

export interface PluginConfig {
  matchers?: WrapWithName<DynamicMatcher>[]
  validators?: WrapWithName<DynamicValidator<any>>[]
  smartEqRules?: SmartEqRule[]
}

Earl can load it via loadPlugin function.

Simple plugin package should run loadPlugin from their main module so the only thing that users has to do is import "awesome-earljs-plugin".

Added validators and matchers will be automatically attached to earl's expect but type info needs to be loaded separately.

Implementation

I started implementing it in #61

• read the whole thing,
• add missing examples,
• solve todos from sourcecode (ensure some conventions for docs)

https://github.com/earl-js/earl/blob/master/packages/earljs/src/matchers/Error.ts#L25

is Error: something, should be [Error: something]

Current limitations / problems:

• If multiple auto-fixes happens in the same file, source maps can quickly get out of sync. First, auto-fix can add multiple lines, and then when the second happens source maps are invalid.
• doesn't work with toBeRejected - looks liked stacktracey doesn't work with async stack traces
• hardcoded stacktrace index to use for caller - quite ugly

This issue facilitates discussions about builtin function mocks design. First of all, we believe that using mocks is so common that making them part of the library is a sane thing to do.

Sinon’s Spy

const spy = sinon.spy(() => 5)

// and for asserts with chai-sinon:
expect(mySpy).to.have.been.calledWith("foo");

What I don’t like about this design is that it doesn’t have support for easy creation of a spy that returns different values for different calls.

Another thing is that I always felt like there is way too many sinon-spy matchers but some really useful were missing.

Jest’s fn.mock()

const mock = jest.fn(x => 42 + x)
expect(mockFunc).toHaveBeenCalledWith(arg1, arg2);

What I like about its design:

• clear design of .mock property exposing all the details - makes it possible to write custom expectations by hand,
• Simple helpers to mock different values for different calls:mock.mockReturnValueOnce(‘x’)

Proposed design

In my experience, I quite often want to return different values for different calls, thus something like Jest’s mock.mockReturnValueOnce (internal queue of returned values) is a must. But this leads us to this weird state where we define returned values in one place but write expectations in a totally another. What if we combined it to something like this:

import {mockFn} from "earl"

const getSizeMock = mockFn([{inputs: [expect.a(Fs), "/path/file.txt"], out: 217}])

This mock expects to be called exactly once with a object of type Fs and string /path/file.txt and returns 217.

There are at least few cool things about this:

• can throw expectations as soon as unexpected call happens,
• inputs is just an array of args so one can use all matchers that would work with beEqual (arrayContaining if you don't care about details, or anything if you really don't care ;d). This API design is just an example, we could utilize helpers like mockReturnValueOnce etc but I wanted to present here that I believe that expectations on a mock can be defined as a part of a mock.

Furthermore, it should be possible to do something like expect(mock).toBeExhusted() to verify if a mock's internal queue of values to return is empty. So there is no need for something like expect(mock).toHaveBeenCalledTimes(5).

I imagine that this could be even further simplified with optional integration with test runner (this needs another proposal) which would ensure all mocks created in a testcase are exhausted after each test run.

There is only one thing that I don't like about this idea - implementing autofix (for inputs) will be challenging as expectations are defined first and execution happens later.

To clarify, this could be preferred way of using mocks but for sure there are cases where more traditional interface like:

const mock = jest.fn(x => 42 + x)
expect(mockFunc).toHaveBeenCalledTimes(3);

is needed and thus should be supported as well.

Thoughts?

For example we have:
expect.stringMatching(...) but not expect().toBeAStringMatching(...)

Hello there,

Disclaimer: This did not end up being a PR as the idea discussed here might be considered controversial. Wanted to talk this through first.

Consider the not modifier. Current implementation is stateful which makes it vulnerable to stuff like:

const expectation = expect(2);

expectation.not;

expectation.not.toBeA(Number); // <-- this breaks

Two things worth mentioning:

• Of course, storing intermediate stuff in a variable is not the preferred way of constructing assertions.
• Earl kinda protects itself against being used in such a manner.

The first obvious step on the way to fixing this seems to be moving from getters to function calls. Then instead of .not you'd have .not(), although:

• Still this falls into the double-negation bucket
• Other assertion libraries seem to have massively adopted the getter pattern (.not)

Finally: How about making .not actually return a fresh instance, immutably?

Pseudocode:

get not() {
    return new Expectation(this.actual, !this.isNegated, this.options)
}

I admit that I haven't tested that thoroughly, although this simple change seems to:

• fix the example above
• let one store intermediate stuff
• lets one do stuff like expect(2).not.not.toBeA(Number) which, although contrived, seems like the behavior I'd actually expect, rather than being able to only negate once.

WDYT?

is void should be never

Add more examples of using given
https://earljs.dev/docs/guides/mocks

Maybe we can just re-export https://www.npmjs.com/package/wait-for-expect or copy the code directly https://github.com/TheBrainFamily/wait-for-expect/blob/master/src/index.ts.

It is great for testing async code and we could have a recipes section that teaches people how to do this.

And we could name the function wait

Hi, we just tried to move to your library from chai. But we were unable to find a matcher for instances. Is there any such matcher yet? How would I write the following chai expect in earl?

const service = new Service();
chai.expect(service).to.be.instanceOf(Service);

Same applies for the property matcher. Sometimes we want to check if an object has a property.

chai.expect(service).to.have.property("prop");

Of course we can circumvent both with equal like

const service = new Service();
chai.expect(service instanceof Service).toEqual(true);
chai.expect(Object.prototype.hasOwnProperty.call(service, "prop")).toEqual(true);

but the method chai offers seems a little bit easier in my opinion.

Also a Regex matcher would be very nice to have. Currently I have to do something like this:

expect(new RegExp(/Error while starting the server.*/).test(msg)).toEqual(true);

which works but the error message is very unspecific being only Error: false not equal to true.

I was just going through the codebase, out of curiosity. Found this:

export function mockFn<FUNCTION_SIG extends (...args: any) => any>(
  defaultImpl?: FUNCTION_SIG,
): Mock<Parameters<FUNCTION_SIG>, ReturnType<FUNCTION_SIG>>
export function mockFn<ARGS extends any[], RETURN = any>(defaultImpl?: (...args: ARGS[]) => RETURN): Mock<ARGS, RETURN>
export function mockFn<ARGS extends any[], RETURN = any>(
  defaultImpl?: (...args: ARGS[]) => RETURN,
): Mock<ARGS, RETURN> {

Two things make me wonder:

1. In overloads 2 & 3, shouldn't (...args: ARGS[]) => RETURN be (...args: ARGS) => RETURN instead?
2. If so, is the third overload even nevessary?

Consider:

declare function mockFn<FUNCTION_SIG extends (...args: any) => any>(
  defaultImpl?: FUNCTION_SIG,
): Mock<Parameters<FUNCTION_SIG>, ReturnType<FUNCTION_SIG>>
declare function mockFn<ARGS extends any[], RETURN = any>(defaultImpl?: (...args: ARGS) => RETURN): Mock<ARGS, RETURN>

mockFn<[x: string, y: number], boolean>() // Mock<[x: string, y: number], boolean>
mockFn((x: boolean) => `${x}`) // Mock<[x: boolean], string>
mockFn<(x: boolean) => string>() // Mock<[x: boolean], string>

Type should be any[] now it's unknown[].

Should earl (or should we develop earljs-utils package) ship with generic test utils?

Utils like:

• clear node module cache - useful for testing modules with state

• ignore files with certain extensions - useful when testing webpack-loadable code with mocha (example: import "styles.css"

• fixture helpers factory - I often use fixture factory function in my tests that allows reusing fixture with small changes between tests

And probably some more...

Should earl be tested using an earlier, stable version of earl? This would enable dropping the dependency on the chai+friends combo.

We should create a plugin suited for smart contract development. It should be highly inspired by Waffle and potentially migrated to the waffle repository in the near future.

Previous @sz-piotr's work: https://github.com/EthWorks/Waffle/pull/408/files

Currently, plugins already can change smartEq behavior via smartEqRules. Problem is that there is no way to change types accepted by toEqual.

Let's imagine that we want to add a smartEq rule that would treat false the same as 0 and true the same as 1. It's not possible currently. Because toEqual always assumes that value is T.

Jest offers a test.each functionality, which can take an array of inputs and run the test over them.

I couldn't find a similar feature in Earl.

Update: is this something that should be handled by the test runner instead? I found this for Mocha: https://github.com/ryym/mocha-each

We need something that will turn tsdoc comments into md.

Node uses inspect.custom to let classes define custom formatting when console logged.

It would be nice for earl to also follow that to reduce verbosity if diffs.

Sometimes you want to fail unconditionally, e.g. because of some runtime condition not being met for successful execution.

Currently, given function type F, a type of a mock of this function is Mock<Parameters<F>, ReturnType<F>>. Wouldn't it be more convenient to write Mock<F>?

Description

Some advanced features like:

• expect a number of assertions
• snapshots

require that earl has some level of integration with a test runner. Such integration can provide:

• hooks before/after test case
• provides a name for a test case

Test runner agnostism

Even though earl should be runner agnostic we optimize for mocha and this should be the first properly supported test runner.

Integration should be as simple as adding earljs/runners/mocha to require option of mocha.

Implementation plan

Good MVP is support for jest's expect.assertions(number). To support this we need post test case hooks. Integration should work with mocha parallel mode.

Old outdated issue: #14

Error messages should clearly point what went wrong and where.

I think Jest does a pretty good job here and probably we can reuse some of the modules.

Ideas:

• excepts should be able to specify context (second arg in Chai)
• Use colors to easily spot differences while comparing objects etc but keep diff "copyable" back to code (chai for example formats objects without commas :wtf:)
• Error message can contain part of the original source with a matcher that failed (jest does it afaik). We already have know how to find original source code (for autofix feature).
• modify stack trace to cut off earls internal. The top stack should be the user's code.

Objects with any non standard prototype will fail currently.

Is there a programmatic interface for the runner?

Use case is to bundle the test file using esbuild for a self-sufficient remote execution (lambda).

Thank you.

Hey! I’m the current lead maintainer of chai, a friend linked me this project, looks cool!

We’ve been slowly extracting useful parts of chai out into separate modules which we hope will help other libraries (like this one) and also help to make chai better.

I noticed you have a couple of open issues around things like a plugin system, and better error messaging, I also notice you’re missing some assertions like deepEqual.

The libraries we’ve extracted so far:

• deep-equal (https://github.com/chaijs/deep-eql) - used for .deep.equal()

• type-detect (https://github.com/chaijs/type-detect) - used for chais .to.be.a() style assertions

• loupe (https://github.com/chaijs/loupe) like nodes util.inspect, used for error messages

• pathval (https://github.com/chaijs/pathval) which converts strings to property lookups, used in the .to.have.nested.property style assertions

• plugin-use (https://github.com/chaijs/plugin-use) which we just released today aims to provide a plugin system which will be the basis of chai 5 which is undergoing a ground up rewrite to use ES modules and typescript.

There’s a few more which have even more specific requirements, but I think some of these might be very beneficial to you.

As an aside, if you feel like you want to contribute to chai, feel free to get in touch. It’s an old project and lots of people use it, so we can’t move as quickly as a new library without annoying/losing lots of existing users, but contributing back to chai means you can impact millions of users!

TODO:

• containerWith - partially matches iterables (arrays/sets)
• arrayWith - partially matches only arrays
• objectwith - partially matches objects

Example:

const container = new Set()
set.add('beer')
expect(container).toEqual(expect.containerWith('beer'))


expect(['apple', 'beer']).toEqual(expect.arrayWith('apple'))

expect({a: 1, b:2, c: 3}).toEqual(expect.objectWith({c: 3}))

Later we can expand arrayWith and containerWith to match multiple values like:

expect(['apple', 'beer']).toEqual(expect.arrayWith('apple', 'beer'))

Jest has toContain and it works with arrays and iterables: https://jestjs.io/docs/en/using-matchers#arrays-and-iterables

Chai has aliases like: .includes, .contain, and .include, which we probably shouldn't try to mimic 😆

Test runner integration (beforeEach/afterEach) /should be implemented to avoid repetition and hard to spot bugs. I propose to not worry about parallel test execution at this stage.

What should happen during integration?

Autoload all earl plugins (with external matchers).

After each test:

• automatically verify all created strictMocks (#12)

Before each test:

• set context for earl like earl.setContext({name: "testSuite/testName"}) That would allow implementation of snapshots without a need to provide a name for it (same interface as jest). I don't want to dive into this now since I am not sure if we want to implement snapshots at all 🤔

How integration should look like

Perfectly it would be as simple as -r earl/integrate-mocha but I am not sure if this can be done like this or we need to force users to add somewhere in there test suite snippet like:

beforeEach(earl.beforeEach)
afterEach(earl.afterEach)

I think earl should ship with integrations for most popular test runners (at least for mocha).

Hi,

in the docs it is described that a promise rejection can be tested with an await like this:

await expect(longTermTask()).toBeRejected('Unexpected error')

However Eslint is complaining about an await of a non-promise (@typescript-eslint/await-thenable). When inspecting I can see that toBeRejected returns void. Either the example is wrong and there should be not await or the toBeRejected function should return a promise.

Note: Both methods are working in my case, removing the await or ignoring the linter rule.