Use test262 format for tests, and submit your tests to test262

[domenic]

If you guys are interested in this, you have some of the better tests in existence, and test262 would benefit from them greatly. Then hopefully stupid things like Safari's non-compliant implementations would not happen, because everybody would be running your tests :)

I'd love to help in any way I can to make this process more smooth, if you are up for it.

I am trying to blaze the trail with my Array.prototype.contains reference implementation: notably the package.json which configures the test runner, and the tests themselves. One big thing to notice is that test262 favors lots of small little files---more or less one for each Mocha it.

I imagine in the process you will find test262 to be pretty anemic compared to Mocha/Chai. In particular we currently do if (!condition) { $ERROR(msg); } instead of assert(condition, message). I think (@bterlson can correct me if I'm wrong) that putting together a basic assert library, containing e.g. assert, assertEqual, assertOwnPropertiesEqual or a few other things, would probably be OK. Maybe a good first step would be to figure out the minimum set of assertions you'd need, and if we agree on them, get them into test262 as helpers.

Also tagging in @smikes who has been doing great test262 work and can probably help advise.

[ljharb]

I've actually filed #48 to switch from Mocha/Chai to tape - I want tests that are simple, and make it easy to do node and cross-browser testing (ie testling), and an npm test command that returns proper unix exit codes.

That said, I'd love to consolidate tests in any way that we can.

[domenic]

Yeah, I guess I was hoping to preempt that move by suggesting you move to the test262 format instead, so that you can share better with browser vendors. The test262-harness project is nascent and we can fix it any way that you want (e.g., not sure if it does proper error codes yet, or how browserifiable it is to make testling or Zuul simple).

[ljharb]

I need a) a console command that returns a zero or nonzero exit code, b) TAP-compliant test output, c) if it needs a harness, it needs to work with testling. I'm open to using test262's format if we can achieve those.

[domenic]

OK, well a) is easy if we don't already have it, b) it already does, and c) might be tricky but might be easy, we'll see.

BTW I don't recommend testling; it's too flaky. I'd go with Sauce Labs (probably via Zuul) instead.

[cscott]

Probably what would be worthwhile in the sort-term is a mocha-test262 bridge, to allow mocha to run test262 test cases. That would let us gradually convert test cases over, instead of having to do it all in one big lump.

[smikes]

Probably what would be worthwhile in the sort-term is a mocha-test262 bridge, to allow mocha to run test262 test cases. That would let us gradually convert test cases over, instead of having to do it all in one big lump.

Yes, I am hoping to build something like that because the python console runner we are currently using for Test262 is pretty limited.

[ljharb]

Awesome, because a python test runner in a JS project is basically a non-starter :-)

[Yaffle]

This is a good topic, because, i think, es5-shim, es6-shim, test262 and other probject should use same tests.

Is it a big difference between mocha or tape or something for ecmascript tests?
Are you serious?

I do not think so, i think testharness should be simple as possible.

One big thing to notice is that test262 favors lots of small little files---more or less one for each Mocha it.

Those small files slow down work really.
Also, it is really hard to change something in test262, as this project is very conservative.

[mathiasbynens]

I agree that the many small little files are not very practical to work with as a test author. But I can see why this is unlikely to change.

What’s worse IMHO is that each file needs to contain a copyright header, so over 50% of the test262 repository is just comments with license information. Can’t we just enforce a default license for all new contributions so that this is no longer needed?

[smikes]

New tests can have the simple copyright header:

/// Copyright 2014 <owner>.  See LICENSE or http://goo.gl/EfffbQ

I can't comment on ECMA's legal concerns with respect to existing code. It may be possible to bulk-replace the licenses for Sputnik and Microsoft-sourced tests with something more compact, but that's a decision well above my pay grade. Maybe @bterlson can comment on that?

(Edited in light of @bterlson's comments, below. I mistakenly thought I was assigning to ECMA, but unfortunately nothing is that simple.)

[bterlson]

Just some thoughts...

@Yaffle

Those small files slow down work really.

Yes, they do :( The rationale, if you're curious: each assert raises an error and stops test execution. More asserts per file mean more hidden bugs when something fails, which means more and slower work for implementers diagnosing failures, and test pass rates are less representative in the presence of failures.

I keep a buffer with the copyright header and a stub frontmatter block and I can crank tests out pretty quickly. I hope tooling will help address some of this too!

Also, it is really hard to change something in test262, as this project is very conservative.

Test262 has different goals and assumptions. Among others, we cannot assume that tests are running on a fully functioning implementation! Test262 needs to remain valuable for implementers bringing up new implementations from nothing. This drastically limits what we are able to do in the test environment (and libraries similar to Chai are likely never to make it into the project despite being very handy).

That said, please file requests you have as issues on the official project! Some may be possible, and others we could figure out something comparable that works. Won't know until you try!

What’s worse IMHO is that each file needs to contain a copyright header, so over 50% of the test262 repository is just comments with license information. Can’t we just enforce a default license for all new contributions so that this is no longer needed?

@mathiasbynens There are two pieces of information we need - who owns the copyright to the test, and what the license is. LICENSE.txt contains the license, but tests need to retain information about who owns the copyright. Granted, this information is already contained in the git commit metadata most likely, so I'll investigate whether we can get away with removing the copyright header.

@smikes Contributors retain copyright of all collateral contributed under tc39/test262-cla so the copyright header should reflect that. Historically, Microsoft has used a separate agreement that assigns copyright to ECMA, hence the number of tests with copyright ECMA.

Copyright header normalization is totally cool (I already did a lot of it with the previous normalization pass, removing most instances of copyright headers with the entire license text pasted in). If there's more you want to do, by all means open an issue!

[Yaffle]

Yes, they do :( The rationale, if you're curious: each assert raises an error and stops test execution. More asserts per file mean more hidden bugs when something fails, which means more and slower work for implementers diagnosing failures, and test pass rates are less representative in the presence of failures.

So the script, which will split tests into multiple files can be used, right? Although contributors will have to run this script anyway...

[bterlson]

@Yaffle I'm not sure I understand your question but if you're asking if you can use a script to convert many-tests-per-file tests into one-test-per-file tests, the answer is yes that's fine...

[cscott]

Any progress on this? I found https://github.com/smikes/test262-mocha-adapter but @smikes work doesn't seem to be complete (and has no readme). There's also https://github.com/smikes/test262-parser which looks useful (but isn't a full solution), and https://github.com/promises-es6/promises-es6 which is in mocha format but doesn't seem to be compatible with test262.

[cscott]

https://github.com/bterlson/test262-harness seems like it's the most mature of the options, but it's not mocha compatible.

[ljharb]

I can definitely set up test262harness for the node tests. However, there's no easy way to run them in browsers yet - ie, no HTML file i can set up that can just be served. That's a blocker.

[cscott]

cscott/babybird@27ca224 is a stab at integrating test262 with mocha for the Promise-related tests. I only found two minor issues (#379) so far.

[smikes]

Thanks- I'm on vacation until dec 8th, will take a look at this when I get back.

The test262-mocha-adapter is at a WIP stage and hadn't been touched for a while, probably not useful.

Test262-harness depends on test262-parser

The "old" Python based tool chain for building the test262 browser tests still exists but it should not be carried forward; it was last used to do the ES5 tests a few years ago. However, some ideas from that process (bundling the tests, compressing them, etc..) might be useful

[cscott]

There's a bugfix patch outstanding to test262-parser which it would be nice if @smikes could merge.

PR #381 has a very rough implementation of running test262 tests inside mocha. But in order for us to run well inside the browser I think we need to actually have a preprocessing step which translates the tests and writes them to files inside our tests directory which mocha can directly run.