# Testing & benchmarks

> The test runners, the release gate, the performance gate, and the benchmark workloads.

One set of focused runners covers fast correctness checks, real-browser behavior, performance
regressions, and the benchmark harness behind the live benchmarks page.

This page is the source of truth for running and maintaining Minnow's tests and benchmarks.

## Choose the smallest runner that proves the change

| Command                        | What it runs                                                          | Use it for                         |
| ------------------------------ | --------------------------------------------------------------------- | ---------------------------------- |
| `npm test`                     | Vitest unit, conformance, differential, and harness tests             | Normal implementation work         |
| `npm run test:coverage`        | The same suite, with the coverage floors enforced                     | Before pushing                     |
| `npm run soak`                 | The generative suites on fresh random seeds, to find new failures     | Hunting bugs rather than pinning   |
| `npm run fixture:format`       | Freezes a database this build wrote, as a compatibility fixture       | Before changing a format version   |
| `npm run test:browser:library` | Core IndexedDB and transaction tests in Chromium, Firefox, and WebKit | Storage or browser-runtime changes |
| `npm run test:browser:site`    | Public-site examples in Chromium, Firefox, and WebKit                 | Docs, examples, and site changes   |
| `npm run test:browser`         | Both browser runners, in sequence                                     | Cross-browser regression checks    |
| `npm run benchmark:gate`       | Seeded Node performance ratios against the checked-in baseline        | Query-executor performance changes |
| `npm run benchmark:sizes`      | Download size of every comparison engine, from the installed packages | Dependency or public-entry changes |
| `npm run check`                | Formatting, types, lint, build, and unit tests with coverage floors   | The local merge gate               |
| `npm run check:release`        | The local gate, performance gate, and all browser runners             | Release candidates                 |

Install the browser binaries once before using a browser runner:

```bash
npx playwright install chromium firefox webkit
```

Everything above also runs in CI: `.github/workflows/ci.yml` on every push and pull request, and
`.github/workflows/performance.yml` nightly. The performance gate is deliberately kept off the
merge path — it measures a machine as much as it measures the code, and a merge gate that fails
on a noisy runner is one people learn to re-run rather than read.

Each browser runner owns its test directory, server, port, and build prerequisites. Shared browser
defaults live in `playwright.shared.mjs`; runner-specific configuration stays in its named
`playwright.*.config.ts` file. This keeps every runner independently callable without hiding which
application it starts.

## What each layer proves

- **Unit and conformance tests** stay beside the source they exercise. They cover deterministic
  behavior, SQL and mutation conformance, differential execution, fault injection, and the
  benchmark generator/oracle contracts. `npm test` picks up `apps/site/bench` alongside
  `packages/**`, so the dataset generator, the oracles, and the suite definitions are checked on
  every unit run.
- **Library browser tests** exercise IndexedDB and transaction behavior that a Node substitute
  cannot prove, and drive a database through a real module worker: the published entry booting,
  transferred buffers arriving intact, and a second worker reopening what the first one wrote.
- **Site browser tests** execute the public examples and drive the benchmarks page itself: one
  case runs a suite end to end in each browser and asserts it verified against the oracles,
  another asserts the `/benchmarks` route is cross-origin isolated. Documentation cannot drift
  into non-running sample code, and the harness cannot drift from the page that runs it.
- **The performance gate** detects regressions on stable seeded query shapes, reads and writes
  alike. It is a guardrail, not a published cross-engine benchmark: the comparison engines run
  without indexes, which is fine for noticing that Minnow got slower and useless as a fair
  cross-engine claim.

When adding a runner, give it one responsibility and make it independently runnable. When adding a
suite case, derive smoke-test counts from the suite itself instead of copying totals into tests or
HTML.

## Seeds, and how a soak failure becomes a permanent test

The generative suites — SQL conformance, DML conformance, and the columnar-versus-row differential
— build their corpora from a seeded generator. A committed run is deterministic: it uses the
checked-in seeds plus every seed that has ever failed, listed in
`packages/core/regression-seeds.json`. That makes the suite a reliable regression net, and on its
own it would be nothing else, because the questions it asks never change.

`npm run soak` is the other half. It runs the same suites on seeds nobody has tried and stops at
the first failure, printing the seed:

```bash
npm run soak -- --rounds 200
```

A failing seed is the whole artifact. Replay it directly:

```bash
MINNOW_SEED=1476318588 npx vitest run packages/core/src/engine/sql-conformance.test.ts
```

Then add it to `regression-seeds.json` under the suite that failed, where every future run picks
it up. Never remove one: a seed in that file is a bug that used to exist, and the entry is what
stops it coming back. The explored space only grows, and it grows by exactly what the soak found.

## Format compatibility

A browser database's data lives in the user's browser, so it outlives every deploy. There is no
migration window and no way to reach back and rewrite it: if a format change makes yesterday's
blocks unreadable, the first anyone hears is a user whose application will not open.

`packages/core/format-fixtures/` holds one frozen database per released format version — a
snapshot, which carries the raw block bytes verbatim, plus the answers that build gave to a fixed
set of queries. `format-compatibility.test.ts` opens all of them on every run, checks the answers
still hold, and checks that writes into a restored database still work.

It also fails when the current `BLOCK_FORMAT_VERSION` or `SNAPSHOT_FORMAT_VERSION` has no fixture
behind it. That failure is the useful one, because it fires _before_ the damage:

```bash
npm run fixture:format
```

Run it on the build that still writes the old format, commit the fixture, and only then change the
version. A fixture can only be produced by the build that writes it — once that code is gone, the
format it wrote cannot be regenerated. For the same reason, never delete one.

## Running out of quota

`quota.test.ts` covers what happens when the browser refuses a write because the origin is out of
space — the characteristic way a browser database fails, and the one an application most needs to
handle deliberately. It pins four things: the write fails rather than half-landing, the error
keeps its `QuotaExceededError` identity so an application can branch on it, everything committed
earlier stays readable, and the same write succeeds once there is room, with no repair step.

## Property-based tests

`block-format/properties.test.ts` states the format's contract as properties and checks them
against generated inputs rather than chosen ones: a value written and read back is the same value,
both codecs agree, a zone map contains everything it summarizes, and a flipped byte is detected
rather than decoded into plausible rows. The generators reach for what breaks encoders — negative
zero, the extremes of the double range, subnormals, empty and astral-plane strings, all-null and
empty columns. A failure shrinks to a minimal case and prints the seed.

## Soaks

Two suites cover accumulation rather than a single operation, which is where a fold that loses a
row or a compactor that stops folding actually shows up:

- `compaction-soak.test.ts` runs 1,500 interleaved mutations against a reference `Map`, compacting
  at checkpoints, and compares the whole table to the reference each time. Compaction is bounded
  and incremental — one call folds a limited number of blocks — so the contract asserted is
  monotone progress, not a fixed point reached in one call.
- `concurrency-simulation.test.ts` runs twelve independent databases over one shared store, the
  shape of twelve browser tabs on one origin, issuing a seeded random schedule. It checks that the
  outcome is explicable: every visible row was written by some tab, acknowledged writes are
  present, no key appears twice, the write the store acknowledged _last_ is the one visible, and
  every tab sees the same database afterwards.

## Concurrent writes

`write-contention.test.ts` documents a sharp edge. Writes commit optimistically and rebase on
conflict, up to `maxCommitRetries` attempts. Issued sequentially, they all land. Issued
concurrently against IndexedDB, at most `maxCommitRetries + 1` land — and that ceiling does not
move with the number of writers, because each commit that wins costs every other in-flight writer
one retry. Sixty-four parallel writes leave the same nine winners as sixteen do.

The tests guarantee the losses are clean: a rejected write leaves nothing behind, an accepted one
is fully present, and which is which is deterministic. An application issuing parallel writes to
one table should await them, or retry the rejections.

## The fault sweep

`packages/core/src/testing/fault-sweep.test.ts` runs one fixed write workload many times, failing
a different storage operation each time — the first block write, then the second, then the first
block read, and so on through every operation a clean run performs. After each interruption it
reopens the store and asks whether what survived is coherent.

The invariant is atomicity, not durability. A write interrupted mid-flight may be present or
absent; both are correct. What is never correct is a torn state — a row nobody wrote, a duplicated
key, a database that will not reopen. Every assertion is phrased as "the surviving state is one a
caller could have observed", never "the write landed".

The set of fault points the workload reaches is pinned in the test. If a change starts or stops
routing writes through one, the sweep fails and someone decides whether it should follow.

## Benchmark workload model

The benchmarks page keeps four workload classes apart. They are never combined into one score:

| Workload | Read measurement                                    | Write measurement                                                |
| -------- | --------------------------------------------------- | ---------------------------------------------------------------- |
| **OLTP** | Selective key, small-set, and bounded-range latency | Point and small-batch inserts, updates, and upserts (1–100 rows) |
| **OLAP** | Scan, join, window, and aggregate execution         | Bulk ingestion and mutation throughput (10,000–100,000 rows)     |

A read workload keeps every query visible rather than collapsing to one figure: each cell is the
median of the timed windows for that query, after one untimed warm-up. Writes keep every operation
and batch size visible the same way. Nothing is reported unless it was verified — every read must
match an independent JavaScript oracle and every write is read back and compared row for row, and
a query an engine got wrong or could not run prints a dash instead of a timing.

There is nothing to publish or regenerate. The [benchmarks page](/benchmarks) has no checked-in
numbers: it builds every engine and runs the suites in the visitor's browser, on the engines,
suites, and dataset size they pick, and explains the measurement methodology, storage differences,
caching, and memory caveats as it reports them. The suites themselves live in `apps/site/bench`
and the page that drives them in `apps/site/app/benchmarks`.

To iterate on the harness, run the unit tests (`npm test`) for the generator, oracles, and suite
contracts, then `npm run test:browser:site` to run a suite in real browsers. To try a change by
hand, `npm run site:dev` and open `/benchmarks`.

## Update the performance-gate baseline

After an intentional executor change, inspect the performance-gate output before updating its
thresholds:

```bash
npm run benchmark:gate -- --update
```

Treat that update as a code change: explain it in review and run the gate again without `--update`.

## Refresh the download-size comparison

The benchmarks page opens with what a browser downloads to run each engine. It is measured from
the installed packages rather than quoted, so refresh it whenever a dependency version or the
public entry point changes:

```bash
npm run benchmark:sizes
```

Each engine's browser entry is bundled and minified with identical esbuild settings, and the
WebAssembly and data files it fetches at run time are added at their shipped size. The result
lands in `apps/site/components/bench/library-sizes.json`, which is a scratch output — nothing
imports it. The figure each engine shows on the page is the `download` field in
`apps/site/components/bench/config.ts`, updated by hand from that run.

---

Minnow 0.1.0 · this page on the site: /docs/reference/testing/