Test Infrastructure

INFO

This article serves as an invitation for sharing ideas to improve our test infrastructure, feel free to contact us on Discord.

In Oxc, correctness and reliability are taken extremely seriously.

We spend a great deal of time strengthening the test infrastructure to prevent problems from propagating to downstream tools.

Parser

Conformance

Parser tests from Test262, Babel, and TypeScript are used to test JavaScript, TypeScript, and JSX syntax.

For Test262, all stage 4 and regular expression tests are included.

All conformance results are stored in a snapshot file for tracking changes:

• test262.snap, reporting 43765/43765 (100.00%).
• babel.snap, reporting 2093/2101 (99.62%).
• typescript.snap, reporting 6470/6479 (99.86%).

All syntax errors are written to these snapshot files for diffing changes.

Fuzzing

To ensure that the parser does not panic when encountering random data, three fuzzers are used:

1. cargo fuzz for sending random bytes to the parser.
2. shift-fuzzer-js by bakkot for producing random but valid ASTs.
3. Automated-Fuzzer by qarmin, which actively reports crashes.

Memory Safety

Oxc uses an arena allocator based around bumpalo as the memory allocator for its AST, and other data. None of the AST node types have a Drop implementation. This is enforced at compile time by Oxc's allocator, which causes a compile-time error if any code attempts to allocate types in the arena which are Drop.This statically ensures that types which own heap-allocated data cannot be stored in the arena, which would result in memory leaks.

Unsafe code

Oxc uses unsafe code for performance optimizations. We aim to contain unsafe to within self-contained data structures which present safe APIs externally. Miri is run on the crates containing these structures on every PR.

Linter

Snapshot Diagnostics

All linter diagnostics are written to a snapshot file for testing against regressions.

For example:

 ⚠ typescript-eslint(adjacent-overload-signatures): All "foo" signatures should be adjacent.
  ╭─[adjacent_overload_signatures.tsx:3:18]
2 │         function foo(s: string);
3 │         function foo(n: number);
  ·                  ───
4 │         type bar = number;
5 │         function foo(sn: string | number) {}
  ·                  ───
6 │       }
  ╰────

Ecosystem CI

oxlint-ecosystem-ci runs oxlint against large repositories to check for false positives, regressions, and panics. The repositories tested include:

• rolldown/rolldown
• napi-rs/napi-rs
• toeverything/affine
• preactjs/preact
• microsoft/vscode
• bbc/simorgh
• elastic/kibana
• DefinitelyTyped/DefinitelyTyped

Idempotency

Idempotency testing is used for integration tests and end-to-end tests on all tools.

An idempotency test follows this procedure:

let sourceText = "foo";
let printed = tool(sourceText);
let printed2 = tool(printed);
assert(printed == printed2);

For example, idempotently minifying a piece of code should yield the same result.

All tools (parser, transformer, minifer, etc.) are idempotently tested on Test262, Babel and TypeScript test files.

Integration Tests

Integration tests are preferred over unit tests.

codecov currently reports line coverage.

End to End

The repository monitor-oxc performs end-to-end tests against the top 3000 npm packages from npm-high-impact.

Its package.json has 3000 dependencies:

"devDependencies": {
  "@aashutoshrathi/word-wrap": "latest",
  "@actions/http-client": "latest",
  "@adobe/css-tools": "latest",
  "@alloc/quick-lru": "latest",
 ...
  "zip-stream": "latest",
  "zod": "latest",
  "zone.js": "latest",
  "zustand": "latest"
}

And a test file that imports these packages and asserts the import:

src/dynamic.test.mjs

import test from "node:test";
import assert from "node:assert";
test("@aashutoshrathi/word-wrap", () => import("@aashutoshrathi/word-wrap").then(assert.ok));
test("@actions/http-client", () => import("@actions/http-client").then(assert.ok));
test("@adobe/css-tools", () => import("@adobe/css-tools").then(assert.ok));
test("@alloc/quick-lru", () => import("@alloc/quick-lru").then(assert.ok));
...
test("zod", () => import("zod").then(assert.ok));
test("zone.js", () => import("zone.js").then(assert.ok));
test("zustand", () => import("zustand").then(assert.ok));
test("zwitch", () => import("zwitch").then(assert.ok));

This test file is run after each of the tools (codegen, transformer, minifier, etc.) rewrites all the files in node_modules.

The packages are updated to their latest versions daily.

This setup has caught many obscure bugs that the conformance test suites missed.

---

If you any ideas on how to improve our test infrastructure, feel free to contact us on Discord.