mck89 / peast Goto Github PK
View Code? Open in Web Editor NEWJavaScript parser written in PHP that generates AST from your code according to ECMAScript specification
License: BSD 3-Clause "New" or "Revised" License
JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification
License: BSD 3-Clause "New" or "Revised" License
Hello, I have been using this library and it works like a charm, my only problem is that I can´t force the rendering of comments. I have been searching through the classes and found that the Base renderer has an attribute called $renderComments, but I don´t know how to use it or even if it can be used.
I have tried the following but with no luck :
Any help is appreciated
Duarte Guerreiro
Hi, awesome project - I will be looking into it.
I would actually like to parse a toy programming language similar to JavaScript for fun; but for that I would need to extend your parser with a few new keywords.
What would be the proper approach if I wanted:
I don’t want to fork your parser; I want to extend it ideally. This toy language is a subset of JavaScript, and I would just validate the AST instead of modifying the parser too much.
Hi there,
it seems that the following code cannot be parsed in this configuration:
$peast_options = [
'sourceType' => Peast::SOURCE_TYPE_MODULE,
'comments' => true,
];
$ast = Peast::latest( 'const HASH_RE = /#/g;', $peast_options )->parse();
hi
The licence suggests bsd-3 composer suggests MIT
could these by synced to the preferred licence
By comparing the minified JS of jQuery and the output of PEAST, I noticed PEAST add emptyy parenthesis for new
operator.
For example, in original jQuery minified file, you have:
return +new Date
WIth PEAST, it becomes:
return +new Date();
It seems these parenthesis are not needed when no arguments are passed to the constructor.
The following currently fails:
$code = '8e-5.toFixed(3)';
\Peast\Peast::latest($code, ['jsx' => true, 'sourceType' => \Peast\Peast::SOURCE_TYPE_MODULE ])->parse();
The issue seems to be the t
of toFixed
, thinking that it's still part of the notation.
Error:
Peast\Syntax\Exception: Invalid exponential notation
lib/Peast/Syntax/Scanner.php:655
lib/Peast/Syntax/Scanner.php:1485
lib/Peast/Syntax/Scanner.php:869
lib/Peast/Syntax/Scanner.php:764
lib/Peast/Syntax/Parser.php:2165
lib/Peast/Syntax/Parser.php:1978
lib/Peast/Syntax/Parser.php:491
lib/Peast/Syntax/Parser.php:197
AST for the above code:
{
"type": "Program",
"start": 0,
"end": 15,
"body": [
{
"type": "ExpressionStatement",
"start": 0,
"end": 15,
"expression": {
"type": "CallExpression",
"start": 0,
"end": 15,
"callee": {
"type": "MemberExpression",
"start": 0,
"end": 12,
"object": {
"type": "Literal",
"start": 0,
"end": 4,
"value": 0.00008,
"raw": "8e-5"
},
"property": {
"type": "Identifier",
"start": 5,
"end": 12,
"name": "toFixed"
},
"computed": false,
"optional": false
},
"arguments": [
{
"type": "Literal",
"start": 13,
"end": 14,
"value": 3,
"raw": "3"
}
],
"optional": false
}
}
],
"sourceType": "module"
}
Peast\Syntax\Exception with message "Unexpected #" throws then try to parse class code with private property like this one:
class ClassPrivateFld {
#msg = 'hello world'
get msg() {
return this.#msg
}
}
The following code cannot be parsed with Peast:
throw new TypeError("Hello\nWorld")
While this works:
throw new TypeError("Hello World")
Getting the following error:
Peast\Syntax\Exception: Unterminated string
AST for the above code:
{
"type": "Program",
"start": 0,
"end": 35,
"body": [
{
"type": "ThrowStatement",
"start": 0,
"end": 35,
"argument": {
"type": "NewExpression",
"start": 6,
"end": 35,
"callee": {
"type": "Identifier",
"start": 10,
"end": 19,
"name": "TypeError"
},
"arguments": [
{
"type": "Literal",
"start": 20,
"end": 34,
"value": "Hello\nWorld",
"raw": "\"Hello\\nWorld\""
}
]
}
}
],
"sourceType": "module"
}
Originally reported at wp-cli/i18n-command#280, it looks like Peast has troubles parsing comments within JSX in some cases.
The following file cannot be parsed, resulting in an Peast\Syntax\Exception: Unclosed {
error:
import React from 'react';
export default class Example extends React.Component {
render() {
return (
<div className="example">
{ /* This is a comment */ }
{ this.props.items.map( ( item ) => {
return <div key={ item.id }>{ item.name }</div>;
} ) }
</div>
);
}
}
In file lib\Peast\Syntax\Utils.php
, the method stringToUTF8Array
can in some circumstances return false
. Indeed, preg_split
can return false
. In such situation, your code output a lot of warnings and crashes randomly after.
It would be cool to add a check and throw an exception. You can get the error preg_split
encountered using preg_last_error
function.
The most common case is badly encoded UTF-8 file (preg_last_error
returns 4); which is the situation I encountered myself : an UTF-8 file with an ISO-8859-1 characters in a comment....
/**
* Converts a string to an array of UTF-8 characters
*
* @param string $str String to convert
*
* @return array
*/
static public function stringToUTF8Array($str)
{
return $str === "" ?
array() :
preg_split('//u', $str, null, PREG_SPLIT_NO_EMPTY);
}
As pointed on #23
Hello,
I have encountered some problems when using this code.
The arrow function does not add the return keyword after the conversion.
Here are the pre-conversion and post-conversion code.
JS:
Pre-conversion code :if(ships.some(s => s.ship.children('.boom').length == 0)){
console.log('111');
}
Post-conversion code:if (ships.some((s) => {
s.ship.children('.boom').length == 0;
//this line should "return s.ship.children('.boom').length == 0;"
})) {
console.log('111');
}
Would it be feasible to add TypeScript support?
Then it'd be easy for me convert typescript to javascript by walking the AST and omitting the typescript only features.
This could be used with a .htaccess rewrite to convert typescript to javascript on the fly, allowing for:
<script src="main.ts"></script>The http request to main.ts would return the transpiled javascript.
That script could cache the generated .js files. On a new request they would only be generated if the .ts file was newer than the generated .js file.
I think this could become a very popular and easy way to serve JavaScript from TypeScript on any web server with php, allowing developers to skip complicated node.js build configurations.
If the parsed code has regular expression literals containing quotes, these quotes are erroneously being parsed as opening or closing a string literal.
This will in turn cause the parser to throw an Unterminated string
error once it hits the end of the line.
As an example, the WordPress Core code contains regular expression literals like this:
function escapeHTML(s) {
var n = s;
n = n.replace(/&/g, '&');
n = n.replace(/</g, '<');
n = n.replace(/>/g, '>');
n = n.replace(/"/g, '"');
return n;
}
In this example, the regular expression literal /"/g
will be parsed as opening a double-quoted string and throw an error once it hits the next line.
Related: wp-cli/i18n-command#98
Hi there,
This isn't a bug report but more of a request for more information.
I'm finding that for large JS files tokenization is taking 20 or more seconds in some cases.
I'm following the example in the docs, e.g;
$source = "large JS file";
$tokens = Peast\Peast::latest($source, $options)->tokenize();
I was wondering if there was any way I could increase the performance of the tokenize method (including possible code improvements and a pull request)?
Any help or pointers would be much appreciated
ElectroMW
An example of the JS that is taking a long time to tokenize;
https://pastebin.com/v7JALVV7
https://pastebin.com/RnEftpiT
Hello,
Using PEAST on the minified version of jQuery v1.12.4 breaks the JS file.
Indeed, no errors occurs during the execution of the PHP code but the reconstructed JS code doesn't work correctly.
Code (where $js_content is the content of https://code.jquery.com/jquery-1.12.4.min.js) :
//Generate the AST
$ast = Peast\Peast::latest($js_content)->parse();
//Create the renderer
$renderer = new Peast\Renderer;
//Associate the formatter
$renderer->setFormatter(new Peast\Formatter\Compact);
//Render the AST
$js_content_new = $renderer->render($ast);
The resulting JS code differs from the original. I think it shouldn't: by this, I mean, each time I use PEAST on a minifed JS, this code outputs the same JS.
The problem doesn't appear if you just include the resulting JS file on a page. For me, the problem appeared when I included the minified JS of jQuery UI 1.11.4.
<html>
<head>
<script type="text/javascript" src="jquery_bad.js"></script>
<script type="text/javascript" src="https://code.jquery.com/ui/1.11.4/jquery-ui.min.js"></script>
</head>
<body>
</body>
</html>
The error you will get is coming from jQuery UI and is:
TypeError: a(...).outerWidth is not a function
Of course, no errors occurs if you use the original jQuery 1.12.4 JS minified file: https://code.jquery.com/jquery-1.12.4.min.js
Libraries like React are very popular and programs using it are mostly written using JSX as well. It would be useful to have it supported in this library.
My personal use case would be having a tool to extract gettext calls from a JSX source file.
I figure this could be done by introducing a new jsx
option on the Parser
class. By default it would be false
.
There seems to be an issue parsing async function declarations, couldn't find an existing or old issue about this.
Given:
Foo = {
async bar() {},
baz: async () => {},
}
Run through:
$ast = Peast::latest($data)->parse();
$renderer = new Renderer();
$renderer->setFormatter(new CompactFormatter());
print $renderer->render($ast);
You would expect:
Foo={async bar(){},baz:async()=>{}};
But instead you get:
Foo={bar(){},baz:async()=>{}};
Steps to reproduce aren't mine, just copying from: https://www.drupal.org/project/drupal/issues/3374660
Fails to parse the following JS:
(function(window, factory) {
var lazySizes = factory(window, window.document, Date);
window.lazySizes = lazySizes;
//if(typeof module == 'object' && module.exports){
module.exports = lazySizes;
//}
})(typeof window != 'undefined' ?
window : {}, function best() { function test() { var a = el[member].test[prop].method(arg).getBoundingClientRect(top); var item = typeof el; if (typeof item === 'undefined') { return true; } }; });
Peast currently fails to parse this code:
$source = "var a=b?.05:-.05";
$ast = Peast\Peast::latest($source)->parse();
PHP Fatal error: Uncaught Peast\Syntax\Exception: Unexpected: 05 in /private/tmp/peast/vendor/mck89/peast/lib/Peast/Syntax/ParserAbstract.php:305
Stack trace:
#0 /private/tmp/peast/vendor/mck89/peast/lib/Peast/Syntax/Parser.php(3423): Peast\Syntax\ParserAbstract->error()
#1 /private/tmp/peast/vendor/mck89/peast/lib/Peast/Syntax/Parser.php(3269): Peast\Syntax\Parser->parseLeftHandSideExpression()
#2 /private/tmp/peast/vendor/mck89/peast/lib/Peast/Syntax/Parser.php(3220): Peast\Syntax\Parser->parsePostfixExpression()
#3 /private/tmp/peast/vendor/mck89/peast/lib/Peast/Syntax/Parser.php(3131): Peast\Syntax\Parser->parseUnaryExpression()
#4 /private/tmp/peast/vendor/mck89/peast/lib/Peast/Syntax/Parser.php(3093): Peast\Syntax\Parser->parseLogicalBinaryExpression()
#5 /private/tmp/peast/vendor/mck89/peast/lib/Peast/Syntax/Parser.php(3049): Peast\Syntax\Parser->parseConditionalExpression()
#6 /private/tmp/peast/vendor/mck89/peast/lib/Peast/Syntax/Parser.php(2911): Peast\Syntax\Parser->parseAssignmentExpression()
#7 /private/tmp/peast in /private/tmp/peast/vendor/mck89/peast/lib/Peast/Syntax/ParserAbstract.php on line 305
{
"type": "Program",
"start": 0,
"end": 16,
"body": [
{
"type": "VariableDeclaration",
"start": 0,
"end": 16,
"declarations": [
{
"type": "VariableDeclarator",
"start": 4,
"end": 16,
"id": {
"type": "Identifier",
"start": 4,
"end": 5,
"name": "a"
},
"init": {
"type": "ConditionalExpression",
"start": 6,
"end": 16,
"test": {
"type": "Identifier",
"start": 6,
"end": 7,
"name": "b"
},
"consequent": {
"type": "Literal",
"start": 8,
"end": 11,
"value": 0.05,
"raw": ".05"
},
"alternate": {
"type": "UnaryExpression",
"start": 12,
"end": 16,
"operator": "-",
"prefix": true,
"argument": {
"type": "Literal",
"start": 13,
"end": 16,
"value": 0.05,
"raw": ".05"
}
}
}
}
],
"kind": "var"
}
],
"sourceType": "module"
}
Same issue for {a:b?.05:-.05}
. But there's no parse error when the dot is not followed by 0 as in var a=b?.5:-.05
. var a=b?0.05:-.05
is working too.
Initially reported in wp-cli/i18n-command#297.
Not really a bug, but I thought it would be worth to mention.
Runinng peast
parser with php
with enabled xdebug
may lead to fatal error, as xdebug thinks we are inside infinite loop and exit the process forcefully.
It happens to me when I parse files with wp i18n
command and my environment happens to have xdebug enabled by default.
Simply prefixing called command with env XDEBUG_MODE=off
works around that issue.
First of all, thank you for this package! It is helping a lot.
I was wondering if it's possible to query not just for values but for key names as well. Something like this:
$ast = Peast::latest($source)->parse();
$ast->query("Property[key='something']");
After running composer install --no-dev --prefer-dist --optimize-autoloader
we noticed the following error:
Generating optimized autoload files
In ClassMapGenerator.php line 73:
Could not scan for classes inside "vendor/mck89/peast/test/Peast" which does not appear to be a file nor a folder
Is this related to the following:
Lines 18 to 23 in bd4c30d
Line 5 in bd4c30d
The test
folder is part of export-ignore
but is registered in the composer.json
autoload
property.
Should it be moved the autoload-dev
property?
"autoload": {
"psr-4": {
"Peast\\": "lib/Peast/"
}
},
"autoload-dev": {
"psr-4": {
"Peast\\test\\": "test/Peast/"
}
},
Peast currently fails to parse this code:
const getTermsInfo = terms => ({
terms,
...(terms === null || terms === void 0 ? void 0 : terms.reduce((accumulator, term) => { // This is line 28847
const {
mapById,
mapByName,
names
} = accumulator;
mapById[term.id] = term;
mapByName[term.name] = term;
names.push(term.name);
return accumulator;
}, {
mapById: {},
mapByName: {},
names: []
}))
});
I'm getting:
Peast\Syntax\Exception: Unexpected: (
Whereas this works:
const getTermsInfo = terms => {
return {
terms,
...(terms === null || terms === void 0 ? void 0 : terms.reduce((accumulator, term) => { // This is line 28847
const {
mapById,
mapByName,
names
} = accumulator;
mapById[term.id] = term;
mapByName[term.name] = term;
names.push(term.name);
return accumulator;
}, {
mapById: {},
mapByName: {},
names: []
}))
};
}
This fails as well
const foo = () => ({ ...(void 0) }); // should return an empty object
This passes:
const foo = () => {
return { ...(void 0) };
};
Formatter/Compact does not introduce a new line character after the single line comment.
This causes the remaining instructions to be ignored and break the execution
this is an example of comment causing this issue
//# sourceMappingURL=
Hi,
The Drupal project is considering adding peast as a dependency to provide PHP-based shortening of JS files, by using the compact formatter to remove comments mainly. https://www.drupal.org/project/drupal/issues/3302755
Before that can be approved by the maintainers we have to perform a standard stability review.
If you could answer these questions it would be very helpful for our due diligence!
Do you have any official policies with regards to:
I couldn't find the security policy on github so a couple additional questions:
I would really appreciate any info you can provide, and please let me know if anything is unclear.
Thanks!
What a great library!
Just out of curiosity i wanted to know if someone has implemented a php transpiler.
The parser seems rock solid and i think it shouldnt be too hard to implement a subset ...
When setting a new value to a StringLiteral
using the setValue
method all unicode escapes are escaped again in Utils::quoteLiteralString
.
Example:
\u0076\u0061\u0072\u0069\u0061\u0062\u006c\u0065\u0073
turns into \\u0076\\u0061\\u0072\\u0069\\u0061\\u0062\\u006c\\u0065\\u0073
The parser does not support dynamic import expressions (e.g. used by webpack)
import('some-file.js').then(_ => {})
Workaround: Before parsing the file, substitute all import\s+(
strings with an alternative token, it will then be exposed as function call in the AST.
Is ES6 converted to ES5 support
In PHP 8.x is seems there's a parse error as the same invoking command works under PHP 7.4
PHP Parse error: syntax error, unexpected fully qualified name "\Syntax\Node\Expression", expecting "," or ";" in phar:///opt/homebrew/Cellar/wp-cli/2.4.0/bin/wp/vendor/mck89/peast/lib/Peast/Syntax/Node/JSX/JSXOpeningElement.php on line 13
Describe the current, buggy behavior
When running wp i18n make-pot . languages/shp_gantrisch_adb.pot
on the command line in a site shell from the Local app on Mac OSX, a very long error message is generated. The initial message is Warning: Uncaught Error in exception handling during call to Error::__toString() in phar:///Applications/Local.app/Contents/Resources/extraResources/bin/wp-cli/wp-cli.phar/vendor/mck89/peast/lib/Peast/Syntax/Scanner.php on line 0
Describe how other contributors can replicate this bug
Run wp i18n make-pot . languages/shp_gantrisch_adb.pot
on the command line in a site shell from the Local app on Mac OSX.
Describe what you would expect as the correct outcome
The POT file be generated without any error messages.
Let us know what environment you are running this on
OS: Darwin 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64 x86_64
Shell: /bin/zsh
PHP binary: /Users/mark/Library/Application Support/Local/lightning-services/php-8.0.22+5/bin/darwin/bin/php
PHP version: 8.0.22
php.ini used: /Users/mark/Library/Application Support/Local/run/_qwoyVNya/conf/php/php.ini
MySQL binary: /Applications/Local.app/Contents/Resources/extraResources/lightning-services/mysql-8.0.16+6/bin/darwin/bin/mysql
MySQL version: mysql Ver 8.0.16 for macos10.14 on x86_64 (MySQL Community Server - GPL)
SQL modes: ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
WP-CLI root dir: phar://wp-cli.phar/vendor/wp-cli/wp-cli
WP-CLI vendor dir: phar://wp-cli.phar/vendor
WP_CLI phar path: /Users/mark/Projects/fse/app/public/wp-content/plugins/shp_gantrisch_adb
WP-CLI packages dir:
WP-CLI global config: /Applications/Local.app/Contents/Resources/extraResources/bin/wp-cli/config.yaml
WP-CLI project config:
WP-CLI version: 2.6.0
Originally reported at wp-cli/i18n-command#280, Peast fails to parse the following JSX using the <>
/</>
shorthand syntax for React fragments:
import React from 'react';
export default class Example extends React.Component {
render() {
return (
<div className="example">
<>
</>
</div>
);
}
}
Errror:
Peast\Syntax\Exception: Unexpected: >
Fragments are being parsed fine if they're on the outmost side though:
export default class Example extends React.Component {
render() {
return (
<>
<div className="example">
</div>
</>
);
}
}
computed = false
, the renderer won't check if property is a legal Identifiera[233] = false;
b['xx-dd'] = true;
$node->setComputed(false);
a.233 = false;
b.xx-dd = true;
Which is illegal and will cause syntax error.
Program:
$src = '{ a: 1 }.a';
$ast = Peast::latest($src)->parse();
Result:
PHP Fatal error: Uncaught Peast\Syntax\Exception: Unexpected: . in .../vendor/mck89/peast/lib/Peast/Syntax/ParserAbstract.php:306
Stack trace:
#0 .../vendor/mck89/peast/lib/Peast/Syntax/Parser.php(201): Peast\Syntax\ParserAbstract->error()
#1 .../tests/test.php(34): Peast\Syntax\Parser->parse()
#2 {main}
thrown in .../vendor/mck89/peast/lib/Peast/Syntax/ParserAbstract.php on line 306
Expected result ($ast pseudo dump):
Array
(
[type] => Program
[body] => Array
(
[0] => Peast\Syntax\Node\ExpressionStatement Object
(
[expression:protected] => Peast\Syntax\Node\MemberExpression Object
(
[object:protected] => Peast\Syntax\Node\ObjectExpression Object
(
[properties:protected] => Array
(
[0] => Peast\Syntax\Node\Property Object
(
[key:protected] => Peast\Syntax\Node\Identifier Object
(
[name:protected] => a
)
[value:protected] => Peast\Syntax\Node\NumericLiteral Object
(
[value:protected] => 1
)
)
)
)
[property:protected] => Peast\Syntax\Node\Identifier Object
(
[name:protected] => a
)
)
)
)
)
Hello,
It seems the compact mode breaks this script:
The output is not working. Replacing with the original code made it working. I will investigate.
Export * as ns
syntax (blocked by for estree/estree#202 to proceed)Peast is used in WP-CLI and so this was experienced when using Composer with WP-CLI.
I thought it might be helpful for the WP-CLI maintainers if I posted an issue here as well since it appears that two (2) of the four (4) notices come from Peast.
The following screenshot illustrates:
P.S. I hope you and your loved ones are weathering the COVID-19 storm successfully. The world is rooting for Italy.
Peast\Syntax\Exception: Unexpected: else
function bug() {
for ( i = 0; i <= 1000; i++)
if (true) {
if (true)
do {
thing()
} while (true);
else 0 !== i ? thing() : i <= 10 ? thing() : thing();
}
}
A return statement which returns a multi line template literal is parsed as a return statement without an argument and the template literal is parsed a single expression afterwards:
$source = "function templateLiteral() {
return `
Hello World`;
}";
$tree = Peast\Peast::latest($source, [])->parse();
$renderer = new \Peast\Renderer;
$pp = $renderer->setFormatter(new \Peast\Formatter\PrettyPrint)->render($tree);
var_dump($pp);
// function templateLiteral () {
// return;
// `
// Hello World`;
// }
Exponentiation operator has right to left associativity.
Program:
$src = 'tmp = 2 ** 3 ** 4;';
$ast = Peast::latest($src)->parse();
$ast dump:
...
Peast\Syntax\Node\BinaryExpression Object
(
[operator:protected] => **
[left:protected] => Peast\Syntax\Node\BinaryExpression Object
(
[operator:protected] => **
[left:protected] => Peast\Syntax\Node\NumericLiteral Object
(
[value:protected] => 2
)
[right:protected] => Peast\Syntax\Node\NumericLiteral Object
(
[value:protected] => 3
)
)
[right:protected] => Peast\Syntax\Node\NumericLiteral Object
(
[value:protected] => 4
)
)
...
Expected result:
...
Peast\Syntax\Node\BinaryExpression Object
(
[operator:protected] => **
[left:protected] => Peast\Syntax\Node\NumericLiteral Object
(
[value:protected] => 2
)
[right:protected] => Peast\Syntax\Node\BinaryExpression Object
(
[operator:protected] => **
[left:protected] => Peast\Syntax\Node\NumericLiteral Object
(
[value:protected] => 3
)
[right:protected] => Peast\Syntax\Node\NumericLiteral Object
(
[value:protected] => 4
)
)
)
...
References:
The following code is removing the block statement when rendered, leading to a syntax error in the final output.
console.log('Start');
lbl: {
console.log('Block Start');
for (let i = 0; i < 5; i++) {
console.log(`Loop 1 ${i}`);
}
for (let i = 0; i < 5; i++) {
console.log(`Loop 2 ${i}`);
if (i === 2) {
break lbl;
}
}
}
console.log('End');
rendered to
console.log('Start');
lbl:
console.log('Block Start');
for (let i = 0; i < 5; i++) {
console.log(`Loop 1 ${i}`);
}
for (let i = 0; i < 5; i++) {
console.log(`Loop 2 ${i}`);
if (i === 2) {
break lbl;
}
};
console.log('End');
The following JS results in an error:
function test() {
function set() {}
return { set };
}
Using:
\Peast\Peast::latest($code, ['jsx' => true, 'sourceType' => \Peast\Peast::SOURCE_TYPE_MODULE ])->parse();
Error message with backtrace:
Peast\Syntax\Exception: Unexpected: }
lib/Peast/Syntax/ParserAbstract.php:305
lib/Peast/Syntax/Parser.php:2639
lib/Peast/Syntax/Parser.php:2816
lib/Peast/Syntax/Parser.php:2762
lib/Peast/Syntax/Parser.php:3618
lib/Peast/Syntax/Parser.php:3271
lib/Peast/Syntax/Parser.php:3199
lib/Peast/Syntax/Parser.php:3150
lib/Peast/Syntax/Parser.php:3077
lib/Peast/Syntax/Parser.php:3039
lib/Peast/Syntax/Parser.php:2995
lib/Peast/Syntax/ParserAbstract.php:349
lib/Peast/Syntax/Parser.php:2971
lib/Peast/Syntax/ParserAbstract.php:213
lib/Peast/Syntax/Parser.php:732
lib/Peast/Syntax/Parser.php:398
lib/Peast/Syntax/Parser.php:359
lib/Peast/Syntax/Parser.php:338
lib/Peast/Syntax/ParserAbstract.php:213
lib/Peast/Syntax/Parser.php:1633
lib/Peast/Syntax/ParserAbstract.php:213
lib/Peast/Syntax/Parser.php:1469
lib/Peast/Syntax/Parser.php:431
lib/Peast/Syntax/Parser.php:357
lib/Peast/Syntax/Parser.php:1982
lib/Peast/Syntax/Parser.php:491
lib/Peast/Syntax/Parser.php:197
AST for the JS snippet above:
{
"type": "Program",
"start": 0,
"end": 62,
"body": [
{
"type": "FunctionDeclaration",
"start": 0,
"end": 62,
"id": {
"type": "Identifier",
"start": 9,
"end": 13,
"name": "test"
},
"expression": false,
"generator": false,
"async": false,
"params": [],
"body": {
"type": "BlockStatement",
"start": 16,
"end": 62,
"body": [
{
"type": "FunctionDeclaration",
"start": 22,
"end": 39,
"id": {
"type": "Identifier",
"start": 31,
"end": 34,
"name": "set"
},
"expression": false,
"generator": false,
"async": false,
"params": [],
"body": {
"type": "BlockStatement",
"start": 37,
"end": 39,
"body": []
}
},
{
"type": "ReturnStatement",
"start": 45,
"end": 60,
"argument": {
"type": "ObjectExpression",
"start": 52,
"end": 59,
"properties": [
{
"type": "Property",
"start": 54,
"end": 57,
"method": false,
"shorthand": true,
"computed": false,
"key": {
"type": "Identifier",
"start": 54,
"end": 57,
"name": "set"
},
"kind": "init",
"value": {
"type": "Identifier",
"start": 54,
"end": 57,
"name": "set"
}
}
]
}
}
]
}
}
],
"sourceType": "module"
}
I'm trying to parse a string which contains this character escaped as UTF-16, but the result is incorrect. I debugged it a little bit and it seems that the issue is due to submitting the UTF-16 units to unicodeToUtf8
one-by-one, rather than decoding the unicode codepoint and then submitting that to unicodeToUtf8
.
Here is a code that reproduces the problem:
// This doesn't work.
echo \Peast\Peast::latest('"\uD83D\uDE00"')->parse()->getBody()[0]->getExpression()->getValue() . "\n";
// These work.
echo json_decode('"\uD83D\uDE00"') . "\n";
echo \Peast\Syntax\Utils::unicodeToUtf8(0x0001F600) . "\n";
echo \Peast\Peast::latest('"\u{1F600}"')->parse()->getBody()[0]->getExpression()->getValue() . "\n";
echo \Peast\Peast::latest('"😀"')->parse()->getBody()[0]->getExpression()->getValue() . "\n";
// This also works (i.e. it shows the smiley face).
console.log("\uD83D\uDE00");
I tested this with PHP 7.4 and the latest master (b33fa0d).
Hi there,
I've come across a failure to parse what appears to be a valid JS file;
Source: https://api-platform.com/component---src-pages-index-js-7905c18163e9101ccbec.js
Command: Peast::latest($src, ['comments' => false, 'strictEncoding' => false, 'jsx' => true])->parse();
I've also come across a failure to tokenize a file that was parsed successfully;
Source: https://api-platform.com/framework-2d6b7d2e2be378715b78.js
Command: Peast::latest($node->render(new Compact))->tokenize();
Any pointers (assuming it is not a bug) would be much appreciated.
Electro
Parsing of bigint literals must be done inside parseNumericLiteral to allow bigint keys in objects.
The following JS code:
if (true) {
const CMS = 'Drupal';
}
parses to:
if (true) const CMS = 'Drupal';
Which result in: SyntaxError: Unexpected token 'const'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.