Test262Harness.Generator 1.2.1

Test262-Harness-dotnet

This is a .NET test runner for Test262: ECMAScript Test Suite. It includes parsing and downloading logic for the test suite in package Test262Harness and test suite generator functionality via CLI too, Test262Harness.Console

Usage

Following projects are utilizing the test suite generation and show how to create NUnit based test suite that is being generated by downloaded snapshot from test262 GitHub repository.

• Jint
  • Generates NUnit test suite to ensure compliance, suite can be run in parallel for faster feedback loop
• esprima-dotnet
  • Generates NUnit test suite for parsing tests, also has custom console logic to compare allow-list.txt for problematic files and progress getting the to parse properly
• acornima
  • Generates NUnit test suite for parsing tests

First you need need to install the required package to your test project, it should look similar to this:

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.9.0" />
    <PackageReference Include="NUnit" Version="4.0.1" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.5.0" />
    <PackageReference Include="Test262Harness" Version="1.0.0" />
  </ItemGroup>

Next you will need a configuration, similar to this (also check the configuration format section):

Test262Harness.settings.json example

{
  "SuiteGitSha": "28b31c0bf1960878abb36ab8597a0cae224a684d",
  "TargetPath": "./Generated",
  "Namespace": "My.Tests.Test262",
  "ExcludedFeatures": [
    "Atomics",
    "Temporal"
  ],
  "ExcludedFlags": [
    "async" 
  ],
  "ExcludedDirectories": [
    "annexB",
    "intl402"
  ],
  "ExcludedFiles": [
    "language/expressions/object/dstr-async-gen-meth-*",
    "language/expressions/assignment/fn-name-lhs-cover.js"
  ]
}

You need to create minimal test file stub to initialize your testing target, example from Jint project.

using System;
using System.IO;
using Esprima;
using Jint.Native;
using Jint.Native.ArrayBuffer;
using Jint.Runtime;
using Jint.Runtime.Descriptors;
using Jint.Runtime.Interop;
using Test262Harness;

namespace Jint.Tests.Test262;

public static partial class State
{
    /// <summary>
    /// Pre-compiled scripts for faster execution.
    /// </summary>
    public static readonly Dictionary<string, Script> Sources = new(StringComparer.OrdinalIgnoreCase);
}

/// <summary>
/// Handles initializing testing state.
/// </summary>
public partial class TestHarness
{
    private static partial Task InitializeCustomState()
    {
        foreach (var file in State.HarnessFiles)
        {
            var source = file.Program;
            State.Sources[Path.GetFileName(file.FileName)] = new JavaScriptParser(source, new ParserOptions(file.FileName)).ParseScript();
        }

        return Task.CompletedTask;
    }
}

public abstract partial class Test262Test
{
    private Engine BuildTestExecutor(Test262File file)
    {
        var engine = new Engine(cfg =>
        {
            var relativePath = Path.GetDirectoryName(file.FileName);
            cfg.EnableModules(new Test262ModuleLoader(State.Test262Stream.Options.FileSystem, relativePath));
        });

        if (file.Flags.Contains("raw"))
        {
            // nothing should be loaded
            return engine;
        }

        engine.Execute(State.Sources["assert.js"]);
        engine.Execute(State.Sources["sta.js"]);

        // initialize engine with Test262 expected host defined functions here
        // https://github.com/tc39/test262/blob/main/INTERPRETING.md#host-defined-functions

        engine.SetValue("print", new ClrFunction(engine, "print", (_, args) => TypeConverter.ToString(args.At(0))));
        // and more...

        // the cinded files that that are expected
        foreach (var include in file.Includes)
        {
            engine.Execute(State.Sources[include]);
        }

        return engine;
    }

    private static void ExecuteTest(Engine engine, Test262File file)
    {
        if (file.Type == ProgramType.Module)
        {
            engine.AddModule(file.FileName, builder => builder.AddSource(file.Program));
            engine.ImportModule(file.FileName);
        }
        else
        {
            engine.Execute(new JavaScriptParser(file.Program, new ParserOptions(file.FileName)).ParseScript());
        }
    }

    private partial bool ShouldThrow(Test262File testCase, bool strict)
    {
        return testCase.Negative;
    }
}

And also the CLI tool for generating the test suite, run this in you test project directory.

dotnet tool add Test262Harness.Console

When everything is installed, you should be able to run:

dotnet tool restore
dotnet test262 generate

Test262Harness.settings.json configuration file

List of most important things you can tweak in configuration file:

Exclusion maps to setting [Ignore] attribute in test suite.

Non-parallel marking maps to [Parallelizable(ParallelScope.None | ParallelScope.Children)] on the affected generated test method(s). NUnit gives such a method its own isolated queue, so the group never runs alongside the rest of the suite, while the cases within the group still run in parallel with each other. That is what a timing-sensitive feature suite such as Atomics.waitAsync needs: those tests assert on elapsed time, and what makes them flaky is competing for CPU with the rest of the run, not running next to one another.

Because all tests under a given sub-directory (e.g. built-ins/Atomics/waitAsync) collapse into one generated method, marking any one entry isolates the whole group. File entries use the test262 forward-slash path format; esprima-style (default) / (strict mode) suffixes are not supported here (the attribute is method-level).

Marking a group no longer serialises its cases against each other. If a group must run strictly one case at a time — because the cases share process-global state rather than merely being timing-sensitive — this setting is not sufficient on its own.

Sharding the generated suite

generate accepts two command-line options for splitting the suite into independent slices so a CI matrix can compile and run them on separate runners in parallel:

dotnet test262 generate --shard-count 4 --shard-index 0

Each test file is assigned to a shard by a stable, order-independent hash of its name, so the assignment is deterministic across runs, machines and operating systems: shard k always contains the same files, the shards never overlap, and their union is exactly the full (unsharded) suite. Both strict and non-strict variants of a file always land in the same shard. Because the assignment does not depend on where generation runs, a shard generated on one operating system can be reused (e.g. from a cross-OS cache) by test jobs on any other.

Branches and releases

• The recommended branch is main, any PR should target this branch
• The main branch is automatically built and published on MyGet. Add this feed to your NuGet sources to use it: https://www.myget.org/F/test262harness/api/v3/index.json
• The main branch is occasionally published on NuGet