The current Babel benchmark should include Babylon parsing just like the Buble benchmark includes Acorn parsing. Thoughts?

PS: This benchmark is a great comparison of existing Javascript parsers, and you can even verify the results from your browser! Perhaps this repository should take a similar approach?

cc: @hzoo @Rich-Harris

Hello!

I understand that uglify-js and uglify-es are devDependencies because they are used in the build.
But they are under benchmarking too, so should not they be dependencies?

It's just a suggestion.

IMO we should wordsmith the README.md a little bit

@sokra mentioned that postcss is a very popular loader for webpack as well, and we should cover that in this benchmark suite.

There seems to be a synchronous API for postcss that just consumes string inputs. We also need to find representative inputs, ideally such that we get a similar runs/sec as the other tests.

See this comment.

Perhaps it should be swapped out with terser, a fork of uglify-es.

Specifically, Mocha seems to perform lots of operations like Array#includes and Array#indexOf on frozen arrays while setting up its environment and executing its tests.

It would be good to have this pattern reflected in the WTB.

Ref. https://docs.google.com/document/d/1X6zO5F_Zojizn2dmo_ftaOWsY8NltPHUhudBbUzMxnc/edit.

This came up a couple of times already (#48 (comment), #50). We should just go ahead and do it.

Hello.

The README says that we can run node dist/cli.js --only babel to run an individual benchmark. But we can't with the current webpack config. I think target: "node" must be set. See https://webpack.js.org/configuration/target/#target

The only way to run a single benchmark with bundled code is by building it with npm run build -- --env.only babel, so npm run benchmark will only run babel benchmark.

Should we edit the README to replace node dist/cli.js --only babel with node src/cli.js --only babel?

GitHub Pages deployment is currently broken due to this Travis CI bug: travis-ci/travis-ci#9312

This is also why the build is currently broken.

It would be great for geomean to have a confidence interval.

Also, the calculation of geomean seems off?

Hi all!

I'm trying to test web-tooling benchmark in my development environment,
but find that it takes too long to run and generates a huge sized file (dist/cli.js).

Is there any lite mode that enables short running time (such as few iterations) ?
Or how can I easily set this mode?

I also build a single benchmark passing --env.only option,
but it seemed that the created cli.js contained all test code like before.
Sadly, single benchmark took a long time as well..

Thanks in advance.

I love this benchmark and use it to compare pc/laptop performance working on web development environment,
do you think it will be a great idea to have submission list of machines that run the benchmark? just like geekbench 5

Hello.

This issue is related to #18.

First, i ran a npm outdated:

@bmeurer, I have some questions about it:

• Should we upgrade all package versions?
• The reason to have explicit version of packages in package.json is the traceability between package versions and project version, right? I mean, having "prettier": "1.8.2" instead of "prettier": "^1.8.2", for example.
• This implies a project version bump, right? 0.4.0?

@sokra mentioned that webpack often uses SourceNode.fromStringWithSourceMap heavily to combine source maps transitively. We currently don't cover that, but should probably include it as well.

We need a predictable mode for the benchmark where each test is run for a fixed number of iterations. Currently benchmark.js doesn't seem to provide this option.

We're accepting PRs that update the frameworks, libraries, and other dependencies in the benchmark.

Hello!

I just published a benchmark generator for web-tooling-benchmark (https://github.com/alopezsanchez/web-tooling-benchmark-generator).

This package generates and modifies the files that you would modify creating the benchmark manually, such as docs/in-depth.md and src/cli-flags-helper.js. It also creates automatically the src/<library>-benchmark.js and src/<library>-benchmark.test.js with a default content, including the license banner.

Demo:

Copied from the project README:

This tool:

• Checks that the user is in in the v8/web-tooling-benchmark repository.
• Checks if the new library already has a benchmark.
• Installs the new library with npm i --save-exact.
• Generates the benchmark and benchmark test files with the naming convention.
• Creates a new section in the documentation file.
• Updates the target list (list of runnable benchmarks).

I think it can be useful to start a new benchmark, saving time to the user.

This is just a Proof of Concept. Use it if you find it useful and give me some feedback if you want 😃
I made it because I think this is a very interesting project and I want to help you to improve it.

Hi, thanks for using virtualfs.
I noticed you're still using the 1.0.0 version. Please note several updates have been made and bugs fixed. I recommend upgrading to the latest. The behaviour as you're currently relying on shouldn't change (I searched your repo for uses of virtualfs).

Often, a change will have an effect on only one part of the web-tooling-benchmark. To run one part in isolation, it's required to edit src/suite.js manually and then rerun npm install. It'd be nice if this were easier. (thanks to @mathiasbynens for walking me through how to do this manually)

We've ran the benchmark in our company on most devices... while it runs predictably on laptops and android phones. We've discovered something's wrong on iPhone X. We get consistently better or similar results to a 7th gen Intel Core i7 on a good part of the benchmarks.
Please check out the screenshots.

i7_7820hq

Pixel running Android P

iPhone X running Chrome

Ref #24 (comment)

These could all similarly apply for babylon itself
FYI we need the bundled/concat'd uncompiled version

the current benchmark:

This benchmark runs the Babel transformation logic using the es2015 preset on a 196KiB ES2015 module containing the untranspiled Vue bundle. Note that this explicitly excludes the Babylon parser and only measures the throughput of the actual transformations. The parser is tested separately by the babylon benchmark below.

Right now it only tests an ES2015 module (albeit a 194kb one 👍) but that may not be representative of what the future will be like so we should think about possible changes to this benchmark:

• Since we have deprecated the yearly presets like preset-es2015 we should run with @babel/preset-env now
  

  web-tooling-benchmark/src/babel-benchmark.js

  Line 12 in 21ca9e9

  options: { presets: ["es2015"], sourceType: "module" }

  • I guess we might want to think about different targets but that just runs less of Babel so not sure how useful that is for a benchmark? (targets: default/ie, current node, current chrome, etc)
• In a similar way I guess we could have a test for an ES3/ES5 file as a good test of the baseline perf of going through the whole program. (The shortcut Babel could do is just to print the file exactly out if it doesn't find any changes, kinda like engines cheat but we won't do that)
  • I just realized we could just run Babel on the output of the original benchmark since that will be ES5 anyway?
• The payload should test out other kinds of code that people are writing/using with Babel like
  • ES2017+ and Stage x proposals (we could use Babel itself for this if we bundled all of it untranspiled, but there are probably other projects we could use)
  • JSX/Flow/Typescript
• There are other things like compiling a minified source but people shouldn't be doing that?
  • Babel operates per file so realistically it compiles a lot of smaller files

https://github.com/babel/minify

After a fresh clone and npm install:

~/src/web-tooling-benchmark master
❯ node dist/cli.js
/Users/ofrobots/src/web-tooling-benchmark/dist/cli.js:52067
/* WEBPACK VAR INJECTION */(function(global) {var scope = (typeof self !== "undefined" && self) || window;
                                                                                                   ^

ReferenceError: window is not defined
    at Object.<anonymous> (/Users/ofrobots/src/web-tooling-benchmark/dist/cli.js:52067:100)
    at Object.<anonymous> (/Users/ofrobots/src/web-tooling-benchmark/dist/cli.js:52129:30)
    at __webpack_require__ (/Users/ofrobots/src/web-tooling-benchmark/dist/cli.js:25:30)
    at Object.module.exports (/Users/ofrobots/src/web-tooling-benchmark/dist/cli.js:90860:69)
    at __webpack_require__ (/Users/ofrobots/src/web-tooling-benchmark/dist/cli.js:25:30)
    at Object.toString (/Users/ofrobots/src/web-tooling-benchmark/dist/cli.js:164851:20)
    at __webpack_require__ (/Users/ofrobots/src/web-tooling-benchmark/dist/cli.js:25:30)
    at Object.<anonymous> (/Users/ofrobots/src/web-tooling-benchmark/dist/cli.js:160010:74)
    at __webpack_require__ (/Users/ofrobots/src/web-tooling-benchmark/dist/cli.js:25:30)
    at Object.<anonymous> (/Users/ofrobots/src/web-tooling-benchmark/dist/cli.js:7374:19)