tc39 / test262 Goto Github PK
View Code? Open in Web Editor NEWOfficial ECMAScript Conformance Test Suite
License: Other
Official ECMAScript Conformance Test Suite
License: Other
Apologies if this isn't the correct place for reporting.
It would be really helpful if the test262 website or wiki mentioned the new Github location of the test262 code. The old Mercurial repository seems to be defunct. Here are a couple of places that mention the old location:
http://wiki.ecmascript.org/doku.php?id=test262:command
http://wiki.ecmascript.org/doku.php?id=test262:submission_process
The only place that I could find that mentioned the new location was in a mailing list message, and this message was pretty hard for me to find.
V8 has support for this (for now): https://code.google.com/p/v8/source/detail?r=24783 --- The test cases I wrote for this might be suitable for adaptation into the ES6 test suite.
There might be a few missing, so it would be good to get a look at it.
This list is from https://bugs.ecmascript.org/show_bug.cgi?id=1453
/cc @evilpie
Current Status is with SpiderMonkey Version: JavaScript-C34.0a1
compiled by me on Windows 7.
All tests pass in SpiderMonkey. Will set up my server and get test result from Firefox.
... and all tests still fail in Firefox.
The following files don't test for the correct value.
We need a higher-level assert library than just using if
s and $ERROR
. This will help those coming from other test frameworks, whether they be shim authors or implementers.
Using Node's assert as a base, here's what's good/bad there:
==
is horrible. In fact we should just use more precise terms like abstractEqual
, strictEqual
, sameValueZero
, sameValue
.throws
are a bit annoying. And in general we prefer negative tests for those. Maybe leave it out initially? If we do include it, I have some ideas for how to make it better.ifError
is not that useful for tests; it's more for callback-based code.fail
is redundant with our $ERROR
.So, initial set:
assert(truthy, message)
assert.sameValue(actual, expected[, message])
assert.notSameValue(actual, expected[, message])
Candidates to consider on an as-needed basis:
assert.throwsWithName(callback, exceptionName[, message])
---literally checks the .name
property of the thrown exception objectassert.throwsExpected(callback, expectedException[, message])
---checks thrown exception is same-value expectedException
assert.arrayLikeEquals(expected, actual[, message])
---checks .length
properties and if they match does a for loop to check samevalue for each element. This is 80% of my use cases for deep equal.What else?
Support request for @smikes, basically.
I added my Array.prototype.contains tests to a local V8 checkout, into which I compiled an implementation. All of the positive tests pass, but all of the negative tests fail.
The command I ran was
$ ./tools/packaging/test262.py --command "/c/src/v8/build/Debug/d8 --harmony" Array.prototype.contains
The failing results look like:
=== es6\Array.prototype.contains\Array.prototype.contains_exception-before-match was expected to fail in non-strict mode
, but didn't ===
--- expected error: Test262Error ---
--- output ---
c:\users\domenic\appdata\local\temp\test262-knixpu.js:62: Test262Error: This error should be re-thrown
throw new Test262Error('This error should be re-thrown');
^===
This kind of indicates the way that it's detecting Test262 errors is off, because they are being thrown, but not detected.
I know there was a lot of work on this recently; is there a chance it broke?
CC @anba
I believe we can make this work just by removing the line which sets negative
to "." Will try this out later.
Some test descriptor entries only appear in a limited amount of test files, maybe these should be removed resp. normalised.
If a test doesn't indicate that it's specifically for sloppy or strict mode it should be run in both. This change should be made at least for the web harness, but also perhaps for the python harness.
Currently tests use a custom syntax for adding attributes to tests. Unfortunately it is difficult to parse and generate and there are no libraries to make this easier. I have a number of attribute normalizations I want to do to make harnessing easier and simplify test authorship, so now might be a good time to change things.
I propose we convert custom format to a YAML Associated List of key-value pairs, with all "flags" moved to a flags array. An example:
/**
* The production Block { } in strict code can't contain function
* declaration;
*
* @path bestPractice/Sbp_A1_T1.js
* @description Trying to declare function at the Block statement
* @onlyStrict
* @negative SyntaxError
* @bestPractice
* http://wiki.ecmascript.org/doku.php?id=conventions:no_non_standard_strict_decls
*/
gets converted to:
/*---
info: >
The production Block { } in strict code can't contain function
declaration;
description: Trying to declare function at the Block statement
negative: SyntaxError
bestPractice: "http://wiki.ecmascript.org/doku.php?id=conventions:no_non_standard_strict_decls"
flags: [onlyStrict]
---*/
JSON has also been suggested over YAML, which I am amenable to except for the excess punctuation and lack of multi-line strings. Some info records are >500 characters so the line length would get absurd.
/*{
"info": "The production Block { } in strict code can't contain function declaration;",
"description": : "Trying to declare function at the Block statement",
"negative": "SyntaxError",
"bestPractice": "http://wiki.ecmascript.org/doku.php?id=conventions:no_non_standard_strict_decls",
"flags": ["onlyStrict"]
}*/
If we can approve this I can move rapidly on the normalizations, including making sure all negative attributes specify the kind of error the expect, moving $INCLUDE's to an includes collection in the front-matter, adding es5id and es6id attributes to track spec sections in our various spec editions (currently only reflected by file path), and etc.
Thoughts?
Test262 has many tests that test the interaction between two features. For example, proxies are generally interesting values and so any API should be tested with proxy values. This has the unfortunate effect of failing those API tests when proxies aren't available, which doesn't make much sense.
One way to solve this problem is specifying a prerequisite.
Two ways I can think of handling this:
specify a function (somehow) that must return true before a test is run
specify prerequisites in the front-matter
a. map feature names to existence checks for said feature
b. map feature names to the collateral for that feature, and only run the test if the tests have a pass rate higher than a certain threshold.
1 is hard to support because we need some way to pass a function from the test to the harness, which isn't something we have today and seems difficult (especially when running tests out-of-proc where you can't just pass the harness a reference to the prereq function).
2.a shouldn't be too hard to support, but since we want to support mapping feature names to test collateral for that feature (#107), 2.b seems somewhat interesting to pursue.
Thoughts?
ES5 has the following to say:
The characters / or backslash \ occurring in the pattern shall be escaped in S as necessary to ensure that the String value formed by concatenating the Strings "/", S, "/", and F can be parsed (in an appropriate lexical context) as a RegularExpressionLiteral that behaves identically to the constructed regular expression. For example, if P is "/", then S could be "/" or "\u002F", among other possibilities, but not "/", because /// followed by F would be parsed as a SingleLineComment rather than a RegularExpressionLiteral. If P is the empty String, this specification can be met by letting S be "(?:)"
The key is that var r = new RegExp(str); eval("/" + r.source + "/")
should give you a regexp identical to r.
Implementations don't follow this too closely. IE doesn't handle empty string properly. FF/Chrome don't seem to handle "\n" properly. There are probably other interesting cases to be discovered.
The following tests assume NaN === NaN
is true
.
Apologies if I've missed it or I'm off track, but I was looking for a test tracking the feature that let
declarations in a for (;;)
style loop are fresh bound per iteration, and I wasn't able to find it. For that matter, I'd also like to see the test for fresh binding for for-in
loops too.
If these tests exist and I missed them, I apologize. I spent about 30 minutes combing through and it became painfully obvious I didn't know what exactly to be looking for.
@mathiasbynens has some rad Annex B string tests over on mathias.html5.org.
It would be great to port them over to test262.
Possibly in the form of a grunt
build system task? I'm suggesting this because policing consistent style is not going to scale across all of the files here. We should just agree on some defaults and let the machine handle the rest.
Can you think of a simple way to support your assert while also showing useful debug information for console hosts, eg. error thrown and line number? I suppose some hosts could define assert such that it throws, and others could report the error and allow the test to continue... But not sure we want hosts to differ in how they execute tests...
Continuing the digression from #22
Yes, I think I can. Rather than using a JSON protocol to communicate about test success, we could use TAP. A test still ends when an exception is thrown and uncaught, leaving error and line number and stack trace information intact for those who can see it. But within the test, the assertions can generate TAP output.
A console runner would write strings to stdout. A browser runner would accumulate strings in an array.
The test harness would need to be augmented a bit to preload the expected number of assertions, but this could be precalculated by the packager and stored in the metadata (or frontmatter, but we don't want to make test authors maintain it.)
So a simple test would produce short TAP output:
1..1
ok 1 - threw ReferenceError on "1 += 1"
A complex test would produce more output:
1..48
ok 1 - message 1
...
ok 48 - message 48
And a failing test, in addition to a throw, would produce truncated TAP output:
1..20
ok 1 - message
not ok 2 - failure message
where the facts that a) an assertion failed and b) 18/20 assertions were not tried would provide finer information about standard conformance than we currently capture.
However this project is bigger than anything I want to start at the moment, so I will leave it at this. Just wanted to document the thought.
Currently the test code is scanned for $INCLUDE()
directives. Inclusion should be marked in the frontmatter instead.
What syntax should be used?
intl402/ch11/11.1/11.1.1_20_c.js needs to be updated to change the expected minor unit value for CLF (Unidad de Fomento) from 0 to 4.
The URL in the test file should also be updated, from "http://www.currency-iso.org/dl_iso_table_a1.xml" to new the location at "http://www.currency-iso.org/dam/downloads/table_a1.xml".
See: http://www.currency-iso.org/dam/isocy/downloads/dl_currency_iso_amendment_157.pdf
See Also: https://bugzilla.mozilla.org/show_bug.cgi?id=946528
There are a number of tests for typed arrays in the Khronos suite. Some of them are out of date (eg. requiring that DataView called as a function should throw), but most should be applicable to the current ES6 draft.
Typed array tests: https://github.com/KhronosGroup/WebGL/blob/master/sdk/tests/conformance/typedarrays
As usual, if porting is going to be problematic for whatever reason we can get coverage from scratch, but a lot of the Khronos tests are fairly in-depth so I'd like to use them if at all possible.
I would like to do this in one of my tests:
Number.prototype[0] = "a";
Number.prototype[1] = "b";
Number.prototype.length = 2;
Do I have to clean up afterward, or can I assume that each test file is independent?
Documenting this in CONTRIBUTING.md is what I am really after :)
The HasProperty check was removed in rev27.
Two tests need to be updated.
I have started work on translating the Promises/A+ tests into a format that works with Test262.
I have a branch for this work -- https://github.com/smikes/test262/tree/promises-aplus-tests-1/test/suite/promises-aplus -- where I created a harness file, a subdirectory test/suite/promises-aplus
for the tests, and a README file documenting the approach that I'm taking.
The first thing I am working out is how to put operations into a particular order without using setTimeout
; the Promises/A+ tests use this at times, e.g., to guarantee that a particular call to resolve
or reject
will happen after all currently queued Promise jobs.
I would be very grateful for feedback about my approach.
Can feature detect browsers by checking for Annex B FIB behavior.
The license/copyright header needs to use single-line comments, otherwise tools/packaging/common.py and tools/packaging/parseTestRecord.py won't work.
Number.toInteger()
was removed from the ES6 draft specification, this test file should be removed, too.
Ref case #74
This code from CONTRIBUTING.md
var error;
try {
[test code]
} catch (e) {
error = e;
}
if (error === undefined) {
$ERROR('expected to throw an error but no error was thrown');
} else if(!(e instanceof ReferenceError)) {
$ERROR('expected to have ReferenceError, got ' + error.name + ' instead.');
}
is frustrating to repeat. Can we turn it into
assertThrows(ReferenceError, function () {
[test code]
});
via a helper?
Consider the case where Promise.all([p1, p2]) is called, and p1 and p2 are both unsettled (non-foreign) promises. Then Promise.all installs an onFulfilled
handler on p1, p2 which is a Promise.all Resolve Element Function
Assuming both p1, p2 eventually are fulfilled, whichever one fulfills second triggers the final steps of that algorithm, specifically:
10 If remainingElementsCount.[[value]] is 0,
a Let valuesArray be CreateArrayFromList(values).
b Return the result of calling the [[Call]] internal method of promiseCapability.[[Resolve]] with undefined as thisArgument and (valuesArray) as argumentsList.
11 Return undefined.
CreateArrayFromList
creates an Array object. If Array.prototype.then
exists and is callable, then instead of fulfilling the Promise with the array, the "foreign thenable" code path in promiseCapability.[[Resolve]] will be exercised, and the Promise returned from Promise.all will be "locked in" to whatever thenable Array.prototype.then looks like.
It's hard for me to think of a good design reason that I would want to add a method then
to Array.prototype or Object.prototype, so this is not something most people will encounter in normal development.
Now this is all obviously insane, and possibly just a minor nuisance, but there may be a security risk here in that an attacker who can modify Array.prototype (which is already quite bad, I understand!) would be able to break any Promise.all resolutions in the same ... Realm? Vat? execution context, maybe.
/cc @domenic
Edited above to clarify meaning.
The following frontmatter parses cleanly:
description: ensure correct value for 'this'
The following causes a test error:
description: 'this' must have correct value
I was surprised by this, and would like to make a note of it here. I believe this is a YAML feature. To avoid problems, don't start scalar fields with a single-quote.
Number.prototype.clz
was replaced with Math.clz32
.
In order to test module loaders and promises, test262 needs to support asynchronous tests.
The most straightforward way would be to extend the signature of runTestCase(fn)
The test case function could get a done
function as a parameter, similar to what Mocha does.
I think this would introduce the least increase to the API surface.
An open issue is what to do with tests that time out, because test262 should not know about setTimeout
.
@domenic @kriskowal would you mind commenting on this?
Following on from: (Clean-up 'simple test harness')[https://bugs.ecmascript.org/show_bug.cgi?id=41]
The simple test harness code in sth.js
and helper.js
is reasonably clean but lacks unit tests and mixes logic with presentation.
I propose to split the code into five separate files -- one each for the four main classes (BrowserRunner, TestLoader, Controller, Presenter) and one for the main
function that sets everything up (anonymous function on sth.js line 682). I would then add unit tests for each of the classes, refactoring when possible.
I would modify packager.py
so that it concatenates these files together into a single file (e.g., test-harness.js
) and make the generated website .html files depend on this new file.
Ultimately a modern build system might be used to concatenate, minify, gzip, etc. etc. all the frontend files. But for the moment, just refactoring and adding tests for existing behavior is enough.
I will start work on this after the #51 PR is disposed of.
While many tests are failing spuriously in light of ES6 changes, we may take some time to update them - current consensus was to update once the spec is ratified, and while I think @anba has captured many of the changes in pull/29, I have yet to review it closely ๐ฆ. However I think we should remove now the tests that freeze new implementations so at least people can still run the official suite (even though the numbers are less meaningful). The only two I am aware of are 15.4.4.5_A4_T1 and 15.4.4.5_A4_T2 (due to toLength change these are no longer boundary value tests). I plan to remove these from the es5 branch next week and update the public website (once I figure out how).
Multi-line comment not properly terminated in es6/ch22/22.1/22.1.2/S22.1.2.1_T3.js.
Running on windows 7 with V8 version 3.28.32 (candidate) [sample shell]
Z:\Code\github\test262>tools\packaging\test262.py --unmarked_default=both --comm
and="v8" 15.5.4.9_CE
=== ch15\15.5\15.5.4\15.5.4.9\15.5.4.9_CE failed in strict mode ===
=== ch15\15.5\15.5.4\15.5.4.9\15.5.4.9_CE failed in non-strict mode ===
Both cause "This application has stopped working" errors; in debugger I get
Unhandled exception at 0x014014e6 in v8.exe: 0xC0000005: Access violation reading location 0x00000000.
Access violation reading NULL pointer at i18n.cc line 537.
I will update my v8 and retest.
Expected test results are wrong in:
Suggested fix:
--- a/test/suite/es6/ch22/22.1/22.1.3/S22.1.3.6_T1.js
+++ b/test/suite/es6/ch22/22.1/22.1.3/S22.1.3.6_T1.js
@@ -10,7 +10,7 @@ runTestCase(function () {
var testArr = new Array('testString', 'anotherTestString', 3),
updatedArr = testArr.fill('newValue', 1, 3);
- if (updatedArr[3] !== 'newValue') {
+ if (updatedArr[3] !== void 0) {
return false;
}
@@ -18,11 +18,15 @@ runTestCase(function () {
return false;
}
+ if (updatedArr[1] !== 'newValue') {
+ return false;
+ }
+
if (updatedArr[0] !== 'testString') {
return false;
}
- if (updatedArr.length !== 4) {
+ if (updatedArr.length !== 3) {
return false;
}
Don't know if this the right place to do this or not, but here goes:
proposal:
since we are betraying the ES grammar/syntax in a mayor way. I will like to introduce a new... more identical familiar way of writing a function:
function()
and that's it.
the ideal came to me from the spite I have from the change of the spec grammar.
Example:
var bar;
bar = [];
[0,1,2,3,4,5,6,7,8,9].forEach(function(arg)bar.push(x));
function foo(i) console.log(i);
Surprising.... this work... in Firefox. But only in firefox it work... not fully sure, but I know it does not work in google chrome.
Will that's it!
Comment, email, or close, you have control.
a note to add this is the same rules following single statment whre:
for()
if()
follow.. if it's on a single line, than the body --open and close curly brace-- can be omitted.
Basically there a bunch of tests that assign to indexes on the global object. This is something that is invalid. https://bugs.ecmascript.org/show_bug.cgi?id=1453
It would be awesome if somebody could remove or modify these tests to not run in a browser environment.
We are seeing inconsistent behavior between major engines over in TreehouseJS/jsdom@15e3c4f
In ES6, array methods can operate on lengths up to 2^53 - 1, instead of 2^32 - 1. This makes writing tests hard. Consider:
if (Array.prototype.contains.call({ length: +Infinity }, 'a') !== false) {
$ERROR('Expected { length: +Infinity } to not contain \'a\'');
}
var arrayLikeWithTrap = {
length: +Infinity,
get 9007199254740992() {
$ERROR('Getter for 9007199254740992 (i.e. 2^53) was called');
},
'9007199254740993': 'a'
};
if (Array.prototype.contains.call(arrayLikeWithTrap, 'a') !== false) {
$ERROR('Expected trapped array-like with length 9007199254740992 to not contain \'a\'');
}
var arrayLikeWithTooBigLength = {
length: 9007199254740995,
'9007199254740992': 'a'
};
if (Array.prototype.contains.call(arrayLikeWithTooBigLength, 'a') !== false) {
$ERROR('Expected array-like with too-big length to not contain \'a\', since it is beyond the max length');
}
all of these tests will likely take longer than the age of the universe to complete, since they are testing an algorithm that takes 2^53 iterations to reach the case we care about.
I'd be curious about other strategies I should employ instead?
The chapter numbers do not correspond between 5.1 and 6.0; for example, in ES5.1 section 7.2 is about lexical whitespace, while in 6.0 section 7.2 is for operators.
Where should ES6 tests go? Probably in their own hierarchy, (as opposed to a branch) so that selected ES5 tests can be applied to an ES6 implementation.
When I reviewed the async test code I believe I saw a provision for overriding the default timeout value by setting the timeout
attribute on a test -- either via @timeout 100
(old-style) or timeout: 100
(new-style).
Someone (me?) should verify that this exists and document it so that it can be used.
Also, some tests may be designed to time out (e.g., a test of a never-fulfilled promise), so there should be a documented way of writing an async test which is expected to time out.
All of this is necessary because we must avoid the use of setTimeout
in Test262 tests.
The test262.py script does not list the 'Expected to fail but passed' list, when --logname is set. It failes with the following error.
Expected to fail but passed ---
Traceback (most recent call last):
File "./tools/packaging/test262.py", line 629, in
code = Main()
File "./tools/packaging/test262.py", line 624, in Main
options.junitname)
File "./tools/packaging/test262.py", line 553, in Run
self.PrintSummary(progress, logname)
File "./tools/packaging/test262.py", line 508, in PrintSummary
self.logfile.append(" %s in %s \n" % (result.case.GetName(), result.case.GetMode()))
AttributeError: 'TestSuite' object has no attribute 'logfile'
It could, e.g., contain a brief FAQ: What is test262? How to contribute? Etc. The content on this page would be perfect; I think it would be worth the redundancy.
These are some errors I see with the current test suite in IE11; just aiming to document them for now. They seem to mainly related to the operation of the delete
keyword/operator.
S8.7.1_A2 Try to delete y, where y is var y=1 Fail
S8.7_A5_T1 Delete referenced object, var __ref = obj Fail
S10.4.1_A1_T1 Checking if deleting variable x, that is defined as var x = 1, fails Fail
S11.4.1_A3.1 Checking declared variable Fail
S11.4.1_A3.3 Checking declared variable Fail
S12.2_A2 Checking if deleting global variables that have the attributes {DontDelete} fails Fail
12.6.4-2 The for-in Statement - the values of [[Enumerable]] attributes are not considered when determining if a property of a prototype object is shadowed by a previous object on the prototype chain Fail
S13_A12_T1 Checking if deleting a function that is declared in global scope fails Fail
15.3.5.4_2-13gs Strict mode - checking access to non-strict function caller from strict function (indirect eval used within strict mode) Fail
15.3.5.4_2-7gs Strict mode - checking access to non-strict function caller from strict function (Function constructor defined within strict mode) Fail
15.3.5.4_2-9gs Strict mode - checking access to non-strict function caller from strict function (New'ed Function constructor defined within strict mode) Fail
15.5.4.9_CE Tests that String.prototype.localeCompare returns 0 when comparing Strings that are considered canonically equivalent by the Unicode standard. Fail
S15.5.5.1_A3 Checking if deleting the length property of String fails
In strict mode, all of these should be passed through verbatim, not boxed.
The following files contain a 'path' entry in the 'flags' sequence. Is that just a leftover from the conversion process?
A number of tests were rewritten to use negative: Test262Error
instead of throwing NotEarlyError
(c33bf0e). I don't quite understand why this change actually works. Applying ToString() on Test262Error
objects returns a string with the prefix "Test262 Error:"
(see Test262Error.prototype.toString
). And applying the requested error pattern /Test262Error/
against "Test262 Error:"
should always fail (cf. error detection in $DONE
).
Am I missing something here?
It would be nice to map a feature name to a set of tests. We could see how many tests exist for each feature and get a breakdown of test pass rates. Would also be nice to pass a feature name to the harness rather than a glob (eg. > test262 Array#contains --prelude containsShim.js
)
Is the file system the best way to do this or should we consider an additional front matter attribute?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.