ScreenBook — testing map

The "don't forget" file: every test layer, what it covers, how to run it, and where the gaps are. Update this whenever a layer is added or a gap closes. Definition of done for any engine change: npm test green AND validate green on the bundled examples.

How to run everything

From a checkout of this repo:

bash

npm i                # once — playwright-core + the TS toolchain (dev-only)
npm test             # builds dist/, then unit + E2E suites (node:test, 74 tests, ~10s)

# full browser matrix on the projects that ship in this repo:
node dist/cli/screenbook.js validate examples/qahwa
node dist/cli/screenbook.js validate examples/tokens-lab
node dist/cli/screenbook.js validate examples/spa-demo

The suites are TypeScript, run natively by node --test (type stripping; Node ≥ 24 for development). They import from dist/ — the suite exercises exactly the files consumers run, never src/ directly. npm test triggers the build via pretest. (Bun 1.3 was verified to run the CLI, the TS source, the MCP server, and the Playwright gate — but Node is deliberately the one supported runtime and gate; 2026-07-19 decision.) Two conventions worth knowing: test files import ./helpers.ts with the real extension (Node does not rewrite specifiers), and a safety setTimeout in a test harness must be .unref()d — one forgotten timer once held the whole suite at 60s wall time.

Docs screenshots (docs/shots/, light + dark variants of every subject) are regenerated in one shot with node tools/capture-shots.mjs — it serves scratch copies of the examples, so nothing in the repo is mutated by the capture run.

Layers

Layer	File(s)	Covers
Kit unit	`test/kit.test.ts`	The REAL `dist/bridge.js` bundle loaded in a Node VM exactly as the shell loads it: deepMerge (arrays replace), scenario overlay (remove→patch order, seed immutability), `{{today±N}}` resolution incl. embedded strings, token flattening, dark-over-light theme merge, fallback safety.
Kit ⇄ bundle guard	`test/kit-parity.test.ts`	Every kit function run through BOTH `dist/kit/` (what Node imports) and the bridge bundle's inlined copy on identical fixtures — proves the bundle ships the same logic it was built from, incl. structural-fake coverage of `cssPath`/`parseMetaFrom`.
Meta contract unit	`test/meta.test.ts`	Meta block parse errors, `withMeta` round-trip, `updated` stamping incl. the manual-override rule, every `lintScreen` violation, `data-go` extraction, filename-prefix order fallback.
Atomic fs unit	`test/fsx.test.ts`	Atomic write (no temp leftovers, parent dirs), change-detection writes, JSON fallback/error paths.
Project API unit	`test/project.test.ts`	Scan/sort, manifest regen stability (no churn), screen CRUD guards (lint reject, dup, missing feature), data/theme/flows shape validation + config registration, review comment filters/resolve, `staticValidate` catching dangling `data-go`, missing/stale manifests, broken flow refs.
Serve E2E	`test/serve.test.ts`	Real HTTP on an ephemeral port: health, `/` redirect, the runtime mount, path-traversal refusal, no-cache headers, review GET/POST → disk, Theme-Lab write-back (`POST /__sb/theme/:id` → file, bad-shape/bad-id 400s), and the live-reload pipeline: raw file write → SSE event → debounced manifest auto-regen.
CLI E2E	`test/cli.test.ts`	Spawns the real binary per command: `init` (full scaffold incl. agent contract + skill, `--force` guard), `manifest`, `validate --static --json` (green → red, exit codes), `serve` (fetches studio/health/screen), `validate` full browser on the init-scaffolded project, `screenshot` (PNG on disk), `tool` — the registry rendered as CLI (catalog, schema help, add-via-`@file` → get → dup-fails → remove round-trip, usage errors, `ok:false` exit semantics), error paths.
Browser matrix	shell `?selftest=1` via `validate`	Every screen × device variant × theme × light/dark × ltr/rtl: bridge handshake, `Engine.ready` fires, meta present, zero console errors (screen AND shell). This is the machine-verified fixed point gate.
App-mode gate	same `?selftest=1` pass	Every `type:"app"` feature: app must register (hello) within 20s, report ≥1 screen, and every reported screen is visited once — zero console errors (driver failures surface as `driver <fn>:` rows; adapters may declare narrow `ignoreErrors` for third-party noise).
Shell interaction E2E	`test/shell.test.ts`	Real Chrome driving the studio against a purpose-built fixture: boot + data binding, live dark/RTL env, `data-go` → deep link, scenario overlay (`remove`) on reload, flow deep links + stepping with pinned scenarios, `Engine.state` across screens + reset chip, review pinning → disk → resolve, Theme-Lab live edit reaching the screen, app mode (embedded cooperative app: screen list, navigation, env follow).
MCP session E2E	`test/mcp.test.ts`	Spawns the real stdio server against an init-scaffolded project: initialize (protocol echo + instructions), 18-tool surface, contract enforcement (lint reject, dup reject), full CRUD incl. `add_app`, static validate through the tool, JSON-RPC error shapes (-32601/-32602).

Projects that double as fixtures

Project	Exercises
the maintainer's host project (not shipped)	iOS frame, AR/RTL/i18n screens, scenarios with `remove`, flows, `statusBar` override, shared.css/js — plus two certified, sealed prototypes embedded as driven apps (31 + 11 screens, drivers injected, app files byte-untouched)
`examples/qahwa`	iOS + Android + web frames in one project, in-screen sheet (EN-24), `Engine.state` cart, feature groups, device filter
`examples/tokens-lab`	Multi-theme (Theme ▾ + theme axis in the matrix), bare `none` frame, live token re-skin, MutationObserver pattern
`examples/spa-demo`	Cooperative app mode — a one-page notes app including `app-bridge.js` itself; engine chrome on an app
`screenbook init` output	The scaffold itself (validated headlessly inside `test/cli.test.ts` on every run); `app add` covered too

Known gaps (candidates for the next hardening pass)

Test bodies are TS but deliberately outside the strict typecheck project (JS-origin code would flood implicit-any); tighten incrementally as suites get touched.
Keyboard navigation (↑↓/jk, /, flow arrows) not yet asserted in the shell suite.
RTL screenshot diffing / visual regression (screenshots exist; no baseline comparison).
Windows: paths are POSIX-tested only; fs.watch recursive + rename semantics unverified.
file:// standalone mode has no automated check (manual only; fallback tokens by design).
Driven-app injection path in the shell suite uses the cooperative mini-app only; real driven-app drivers are exercised in the maintainer's host project, not by npm test.