https://www.npmjs.com/package/json-schema-to-ts

https://github.com/micnic/r-assign

https://github.com/samchon/typescript-json/tree/master/test/structures

I'm developing a runtime type checker (validator) library, and have prepared lots of test cases, for hardcore validation.

How about typescript-runtime-type-benchmarks to adapt such test cases?

They may be useful.

Originally this project was started with libraries that had support for constraint checking (e.g. minimum number or max string length).

But I had removed it in 61b353c:

• to allow for more libraries to be included that only had type checking capability
• because I did not like the way it was designed and wanted to redesign it later

This issue is an open discussion for finding a common denominator for most packages that we could benchmark. A set of functionality that is widely supported, yet is not too shallow.

If I read the docs of ajv correctly, it only offers type inference for JSON Type Definition (using the JTDDataType type), but not for JSON schema.

The test case in this repo is using JSON schema though, and as such, AFAICT it does not meet the criteria of this benchmark, specifically that it does not cast to the right type.

https://github.com/richardscarrott/ok-computer

Not sure if it fits with the spirit yet, but need to take a look.

https://github.com/sinclairzx81/typebox

Hi,

Nice project, thanks, i've used it for my evaluations.

I've noticed the huge gap between 2 libraries to all other libraries

• ts-quartet
• ts-json-validator

This huge gap is probably because of the way the project is running the tests.
The 2 libraries above use a different strategy than all others to create the validators.

While others mostly use predefined, hard-coded validator functions and through composition of them create a schema, the fastest 2 libraries will compile JS code at runtime (eval() or new Function(...)) to create discrete validation functions that do not call other functions internally (no composition) but instead have all the required validation code within the same function created specifically for the schema.

For example, Quartet:

For the following schema:

const checkData = v<Data>({
  number: v.safeInteger,
  negNumber: v.negative,
  maxNumber: v.positive,
  string: v.string,
  longString: v.string,
  boolean: v.boolean,
  deeplyNested: {
    foo: v.string,
    num: v.number,
    bool: v.boolean,
  },
});

It will generate the following validator function:

function validator(value) {
  if (value == null) return false
  if (!Number.isSafeInteger(value.number)) return false
  if (value.negNumber >= 0) return false
  if (value.maxNumber <= 0) return false
  if (typeof value.string !== 'string') return false
  if (typeof value.longString !== 'string') return false
  if (typeof value.boolean !== 'boolean') return false
  if (value.deeplyNested == null) return false
  if (typeof value.deeplyNested.foo !== 'string') return false
  if (typeof value.deeplyNested.num !== 'number') return false
  if (typeof value.deeplyNested.bool !== 'boolean') return false
  return true
}

This has a deep impact on performance depending on how you run your code.

The benchmark code in this project will use 1 schema and iterate over it for a certain period of time. This is perfect for quartet because of how V8 works.
The function becomes super hot, it quickly becomes inlined and additionally, if any internal function call exists within the validator it will get inlined as well!

In other libraries, this can not happen because so many functions are called, due to the composition, so most of them are cold and nothing get's inlined.

In real world scenarios, such a perfect order does not exists. For example, when handling incoming request, so many functions are called that by the time we reach the validator it is no longer hot!
And of course, we also need to factor in handling of multiple incoming requests.

The major advantage of the 2 libraries in question does not play along in real world scenarios and the results of the 2 are distorted in the benchmark.

I should also note the security risks of using runtime code evaluation. For a popular and heavily used library like ts-json-validator (which is actuall ajv) this is less of a concern. For a almost not used, not popular library like quartet I will take caution.

I general sucj a huge gap does not make sense, otherwise everyone would have used these libraries entirely.

Thanks again!

https://github.com/jeengbe/vality

@jeengbe Want to give this a try? 😁

Hi, we use a lot of validation in browser runtime with "joi" library. Could you please add some benchmarks about this package?

https://github.com/5alidz/tiny-schema-validator

https://github.com/vriad/zod

Currently, only data types itself are tested. This has been introduced in 61b353c. Validation benchmarks without basic data content validation is way too shallow to be called a validation benchmark. At best it is a type guard, but that has nothing to do with data validation. Almost all real-life validation code does use features like enums, negative number checks, string length limits, or email checks, etc. If this is not part of this benchmark, then this benchmark is highly misleading.

If a validator library is not able to validate the actual content for something like number ranges, or string length etc, then it should not be called data validator and not be part of this benchmark suite.

If you focus only on type guards only, without data content checks, then please add this to the README. Then we can remove the benchmark links from Marshal since comparing apple with oranges it the last people want.

Please

https://github.com/arcanis/typanion

Yarn 3 uses it

It seems that there is a huge gap between the results of different packages in this benchmark, by going deeply in the implementation of the parsing process of packages that perform better I saw that for example @badrap/valita does not return a new object and does not strip properties that are not part of the schema, this gives an advantage of about 4 times in some tests that I made with my own package.

It seems that more mature packages do return a new object with the properties copied from the provided object by default, while newer packages tend to return the same value that is provided, which give them some advantage.

I propose to add a criteria for the benchmark results to expect the same object or a new one, this way the unfairness of the results will be minimal.

https://github.com/ai-labs-team/ts-utils

Hi, great project!

Would it be possible to include AJV to this test suite? I think a case exists for testing AJV performance as it's possible for TypeScript to inference types from JSON schema directly via conditional type mapping. Would certainly be curious to see how the defacto JSON schema validator holds up against these libraries.

Many Thanks!

https://github.com/samchon/typia

I don't know if there is a way to specify the dependency on the GitHub actions to be completed.

But currently GH pages deploy in parallel, and before any benchmarks are done.

Ideas?

Ref: #897 (comment)

https://github.com/ar-nelson/spartan-schema

@ar-nelson Wanna contribute? 😉

Having all validator packages installed in one Node package (project) may lead to wrong results, and even errors, if dependencies of validators are shared, but have loose versions defined.

The potential solution is to convert this to a monorepo, which has a separate package for each validator.

But the ideal solution should not break dependabot / revonatebot.

@hoeck thoughts?

I made isValid much more performant in 6.0.1, it should be more on par with the other faster libraries. checkValid and validate both produce errors arrays that cost memory/CPU to construct.

Also, thanks for maintaining this benchmark!

None of the graphs on the README seem to have any data on them.

Using projen would open up the possibility of having different configs for each package that is being tested.

This would be necessary for #898, but we have also ran into this with spectypes (#856).

• node_modules Isolation #1219

As a part of the Deepkit framework they provide a runtime type implementation documented here: https://deepkit.io/documentation/type

From a suggestion by @shlomiassaf, in #300

Create a vanilla JS or TS validator.

Parallelization had to be disabled a while ago, because the final step of the benchmarking run (per Node version) is to commit back the results. This resulted in a race condition, between runs and some would fail to commit, as they had a stale git checkout.

I think we can fix this by running the benchmarks in parallel, but instead of committing the results right away, use GitHub actions artifacts mechanism to store the results in the cache.

Then as the last step, get the cached results at once, and commit them at once.

Hi, thanks for providing these benchmarks and for collecting all those libraries.

I'd like to add my own runtypes lib to it: https://github.com/hoeck/simple-runtypes - would you accept a PR for that?

https://github.com/jsoldi/to-typed

@jsoldi? 😁

https://github.com/mojotech/json-type-validation Please add benchmarks for this package. Happy to help if anyone can give some info for adding the tests.

https://github.com/fabiandev/ts-runtime

https://www.npmjs.com/package/type-safe-validator

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• chore(deps): update dependency @types/node to v18.19.31
• fix(deps): update dependency @badrap/valita to v0.3.8
• fix(deps): update dependency @deepkit/core to v1.0.1-alpha.145
• fix(deps): update dependency @deepkit/type to v1.0.1-alpha.146
• fix(deps): update dependency @deepkit/type-compiler to v1.0.1-alpha.146
• fix(deps): update dependency @mondrian-framework/model to v2.0.40
• fix(deps): update dependency @sapphire/shapeshift to v3.9.7
• fix(deps): update dependency fp-ts to v2.16.5
• fix(deps): update dependency purify-ts to v2.0.3
• fix(deps): update dependency serve to v14.2.3
• fix(deps): update dependency superstruct to v1.0.4
• fix(deps): update dependency ts-node to v10.9.2
• fix(deps): update dependency vality to v6.3.4
• chore(deps): update babel monorepo (@babel/cli, @babel/core, @babel/preset-env, @babel/preset-typescript)
• chore(deps): update dependency @types/ts-expose-internals to v5.4.5
• chore(deps): update dependency @types/yup to v0.32.0
• chore(deps): update dependency expect-type to v0.19.0
• chore(deps): update dependency gts to v5.3.0
• chore(deps): update dependency ts-patch to v3.1.2
• fix(deps): update dependency @sinclair/typebox to v0.32.28
• fix(deps): update dependency ajv to v8.13.0
• fix(deps): update dependency myzod to v1.11.0
• fix(deps): update dependency preact to v10.21.0
• fix(deps): update dependency reflect-metadata to v0.2.2
• fix(deps): update dependency rescript to v11.1.0
• fix(deps): update dependency rescript-schema to v6.4.0
• fix(deps): update dependency ts-runtime-checks to v0.5.1
• fix(deps): update dependency typia to v5.5.10
• fix(deps): update dependency unknownutil to v3.18.0
• fix(deps): update dependency valibot to v0.30.0
• fix(deps): update dependency vega to v5.28.0
• fix(deps): update dependency zod to v3.23.6
• chore(deps): update dependency @types/node to v20
• fix(deps): update dependency rulr to v10
• fix(deps): update dependency svgo to v3 (svgo, @types/svgo)
• fix(deps): update dependency typia to v6
• fix(deps): update dependency yup to v1
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency @types/svgo to v2.6.4
• fix(deps): update dependency mol_data_all to v1.1.1097
• fix(deps): update dependency parse-dont-validate to v4.1.2
• fix(deps): update dependency succulent to v0.20.0
• fix(deps): update dependency typescript to v5.4.5
• fix(deps): update dependency vega-lite to v5.18.0
• fix(deps): update dependency csv-stringify to v6
• fix(deps): update dependency decoders to v2
• fix(deps): update dependency ok-computer to v2
• fix(deps): update dependency suretype to v3
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• chore(deps): update dependency tsconfigs to v5

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

https://github.com/brielov/typed

https://github.com/yona3/typescanner

Due to a breaking change introduced in TypeScript 4.1, Promise.resolve() now requires an argument. The most recent builds have been failing due to an offending line in index.ts (shown below):

https://github.com/ryansmith94/rulr

Maybe @ryansmith94 can commit a PR? :)

Need to add contributors section to the README

They seem to run for all PRs, which is wasteful and unnecessary.

PRs should run tests only.

Hey, thanks for this benchmark! It's a great idea.

I think right now the existing test cases aren't testing for the same behavior in each library though. Depending on the library, it might support "guarding", "validating", "parsing", "transforming", etc. But if you compare guarding of one to parsing of another it's not a direct comparison.

For example, as far as I can tell right now toi is simply guarding a value and returning a boolean in its test case, whereas superstruct is parsing an input, cloning it, and returning the parsed value in its test case (instead of using the simpler is export).

It might make sense to add more specific levels of functionality to each test case. And some libraries simply won't allow certain functionality. For example (using superstruct because I'm the author so I know it best):

import { is, assert, coerce, object, ... } from 'superstruct'

const type = object({ ... })

class SuperstructCase extends Case {
  name = 'superstruct'

  // The `is` test is for simply returning a boolean indicating whether 
  // a value is valid. It doesn't need to create errors, parse values, etc.
  // This is the most basic and the minimum to be included.
  is(value) {
    return is(value, type)
  }

  // The `assert` test is for throwing a proper `Error` object, with a 
  // reason for why validation failed. It shouldn't be re-implemented
  // for libraries that only support `is` type guards because the errors
  // won't actually contain any information.
  assert(value) {
    assert(value, type)
  }

  // The `parse` test is for libraries that allow some parsing step on
  // the value to ensure that it is valid—for things like default values,
  // coercing inputs, etc. It should return the parsed value, and the
  // value *should not* equal the input (ie. cloned).
  parse(value) {
    return coerce(value, type)
  }
}

You'd be able to show a section of graphs for each level of functionality. Then people can make more informed decisions—eg. a library might be much faster in is, but not allow for parse.

I think this would make the comparisons much more accurate.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

The SVGs in the readme do not reflect the repo due to the cache. Consider updating automation to add a random value to the URLs in the readme after each update.

Create a table / matrix for each package and features it supports.

https://github.com/Olian04/Recordari

Hi, thanks for this project.

I'm currently looking at submitting a new runtime type checker for comparative performance benchmarks, but have noticed that the order in which each validator is run seems to degrade subsequent benchmark results. By moving performing tests around, I'm able to greatly improve the performance results of tests by simply running them first.

I'm thinking this is likely due to some of the validators utilizing more memory overhead (implicating the GC), or that V8 optimization heuristics have broken down causing V8 to take the slow path (this is quite likely given how validation routines may require dynamic property access on objects to complete). To rule out the GC, I started the project with npm run compile:spectypes && node --expose-gc -r ts-node/register ./index.ts and called global.gc() to force collection prior to tests running, but no luck. So the problem is most likely related V8 optimization heuristics.

I think to give all tests an even footing, I'm wondering if it would be possible to reshape these tests such that each is run within it's own node process (rather than running all tests in the same process). This would give each a fresh instance in which to do it's thing, and should yield more accurate results.

Open to thoughts if this is something worth looking into.