thelarkinn / js-parser-discussions Goto Github PK

Discussions & Collaboration on a Unified/Base set of parser features for JavaScript

js-parser-discussions's Introduction

js-parser-discussions

Discussions & Collaboration on a set of parser features for JavaScript for interoperability.

Disclaimer

This is a blameless medium. Focusing on a specific toolings shortcomings or bias or personal agendas, will not help accomplish the following goals. Therefore I urge that anyone involved uses an unbiased inspection of these tools to help identify ways to create an unopinionated, and extensible foundation.

Synopsis

Currently there are a variety of JavaScript parsers that are available. Each of these parsers have specific purposes. These parsers include (but is not limited to):

Family of ESTree:
- SpiderMonkey parser
- Esprima
- Acorn
  - Espree
- Babylon
- Shift
UglifyJS (no dependant parser)
UglifyJS2 parser
Typescript parser
Traceur parser
Flow parser (ml based)

Each of these parsers have specific feature set. For this synopsis, the reasonings behind those featuresets are irrelevant, however it is noteworthy to mention that often specific parsers are companions to specific frameworks, libraries, toolings, etc.

Features?

Esprima
Acorn
Babylon
Uglify

Current Challenges

Currently, frameworks, libraries, tools, optimizers, bundlers, coverage mappers, source-mappers (?), instrumenters, testing utilities, etc. have to make the choice to use 1 of the following parsers.

This causes many issues for interoperability:

istanbul/coverage + transpiled libraries
webpack + acorn + limited type support
program flow based optimizations have to be implemented differently for every parser

Naive Proposal/Discussion

What if we think of these different parsers as simply a set of features for a single base parser? This could unify tools, parser consumers and promote interoperability. A rich and configurable feature/plugin system, then allows for multi-framework support, and extensibility. (Experimental use of ES features, etc.)

parse
  |-- event delegation
  |-- syntax flagging
  |-- custom estrees(?)/AST's
  |-- typed language support
  |-- (?)

Lessons Learned from:

Acorn

Plugin system is hard, complex. Where can we learn from this?
Event delegation drives important bundlers and tools like webpack. If this is a feature, should it be written in a way to interop with other plugins features? IE: extended, transpiled languages (events for TS, Flow, CJS, etc.).

Esprima

Limiting features can yield higher parse performance. Where is this relevant, can this help establish a base

Babylon/Babili

Program Flow is important and can be leveraged for optimizations, compression, minimizing. However the level of optimization is based on the amount of knowledge known (types, etc.). Can this be applied to more than just ES6?

js-parser-discussions's People

Stargazers

Watchers

Forkers

rreverser

js-parser-discussions's Issues

Key Differences Between Acorn / Esprima

I found this quote from nicholas zakas pretty interesting about why eslint couldn't use acorn directly:
https://github.com/eslint/espree#frequently-asked-questions

Acorn is a great JavaScript parser that produces an AST that is compatible with Esprima. Unfortunately, ESLint relies on more than just the AST to do its job. It relies on Esprima's tokens and comment attachment features to get a complete picture of the source code. We investigated switching to Acorn, but the inconsistencies between Esprima and Acorn created too much work for a project like ESLint.

We are building on top of Acorn, however, so that we can contribute back and help make Acorn even better.

This repo should try to highlight clearly what differences exist between the various parsers, most of all of which share the same AST structure and basic API.

Create separate Markdown file which lists all current parser features

From #1

I think it's useful to have a section (perhaps starting with a new .md file) that lists every parser feature and its commonly agreed definition. This does answer the question "What does it mean to have a syntax feature flag?".

Possibly Clarify Esprima-based Parsers vs. Others

There is a core group of parsers that basically follow the API and AST from Esprima. When people talk about JS parsers I pretty much just think about this fairly short list:

espree
esprima
acorn
babylon*

And then there are other parsers of various kinds that aren't really compatible for various reasons. You mention uglify, typescript, flow and the others I assume would fall into this category. Is sweetJS even a parser? I think you should separate these two classes of parsers in some way. Either only focus on the esprima-inspired parsers or lump them together in some way, because they have the same base APIs, etc.

Let me know if astxplorer can help in any way

This is not an issue, but I wanted to offer my help. If you think https://github.com/fkling/astexplorer can add value to this project, please let me know. Maybe having a look at the source can be helpful, especially how we abstract the different parser APIs. Or maybe we could extend astexplorer to provide some kind of comparison of parsers.

And if not, that's OK too 😉

Best way to format this repository

This discussionary/proposal repo is kind of a new thing for me. I've added @RReverser @ariya and a few others as collaborators to help edit (if they are interested) the initial layout.

This can mature, but I think a format should be found to stick to for:

Discussion points
Implementation details
etc

Features: CST element collection

I have a proof of concept for post processing babylon's ast to add the CST-in-AST nodes specified in estree/estree#107 at https://github.com/forivall/tacoscript/tree/master/packages/cstify.

There's also a much more robust implementation at https://github.com/cst/cst

I'm not sure if this should be in features, or in a separate CST area.

Better list of current challenges

Since my experience is limited I don't have that many things to put under "Current challenges". I however, have heard of, or imagine there has been more that what is currently listed. We'll have to take some time (when there is some time 😅 ) and add a few more.

API Examples

Can you provide some explanation of how to use these various parsers?

Essentially the APIs for most of them are the same.

Highlight Dependencies

Which projects are using which tool?

In my head everyone has pretty much converged around acorn, but maybe babylon will become the new standard. I have a fairly limited set of tools I work with however, so it would be nice to see a list of modern projects and which parsers they use.

eslint for example uses espree, but mostly as a simple translation layer between acorn and esprima for error handling and other things, since eslint originally used esprima and now uses acorn under the hood (via espree).

Investigate Uglify

A lot of modern tools use an esprima/estree-compatible parser.

Can we find out how many projects are using uglify and if it should be included in this discussion.