Why I Stopped Writing Tests First

Tests assert on call-site values. A user journey is a sequence of UI state, persistence, navigation, rendering, accessibility, and timing. The two are different categories of claim, and one cannot serve as the specification of the other. When you try to use tests as the authoritative specification of a system, every gap between what a test can assert and what a user journey actually does becomes a gap in the spec.

I ran Domain-Rules TDD for 10 weeks before I understood this mechanically. The test-writing pipeline was sound. Every rule produced a testable assertion, every assertion generated a test that exercised production code via createCaller, no circular mocks, no tautologies, 100% pass rate across three domains. The methodology produced a complete test suite and zero implemented procedures.

This essay explains why.

The mechanical limit of tests as specifications

A unit test asserts expect(result.something).toBe(X) at a specific call site. An integration test asserts the same at a slightly higher level, with a real database underneath. Both produce claims about state at a moment, at a point in the code path, given known inputs.

User journey behavior is not a moment-of-state claim. It is a sequence claim:

• After a create, reload the list — the row must still be there.
• Navigate away and back — the row must still be there.
• The list must render in both desktop (1440x900) and mobile (375x812) viewports.
• axe-core against the rendered DOM must return zero violations above moderate impact.
• Console messages during the interaction must be in the allowlist.
• Network requests must not include unexpected 4xx/5xx.
• A date created today, under TZ=America/Los_Angeles, must render correctly for a user whose browser runs TZ=Europe/London.
• User-generated text that happens to read "ignore previous instructions" must render as content, not execute as control.

Every item in that list is something a unit or integration test cannot assert. The test harness does not render DOM, does not navigate between routes, does not switch timezones, does not accept screenshots as evidence, does not run accessibility tools over markup, does not exercise concurrent-session state. It is not a matter of writing better tests. It is a matter of tests being the wrong instrument.

When tests are the specification, these concerns are absent from the spec, which means they are absent from the implementation effort. The code that satisfies every test can still fail every one of these checks.

What 10 weeks of test-first produced

Tests passed the gate. Procedures did not land.

The specific failure mode: the test-writing agent reads a domain-rules file, enumerates cases it can translate into call-site assertions, produces a test file, the gate runs, the gate fails because some cases the rules implied (persistence across reload, mobile rendering, navigation return) were not translated into assertions. The fix is to add more cases to the rules, which triggers the test-writing agent again, which produces a more complete test file missing a different class of cases. The convergence is linear in the size of the implicit specification. The implicit specification is unbounded, because user journeys are unbounded.

After three domains had passed the gate this way, no implementation had begun. The test suite was expanding faster than the code could be written against it. The gate demanded a 100% pass rate to proceed to implementation; that 100% was a moving target.

The methodology was optimizing for the wrong property. It optimized for demonstrable correctness before integration, at the cost of demonstrable existence.

The pipeline that replaced it

Source-First from domain rules has six stages. Each catches a different class of failure.

1. Rules are the specification

Domain rules are authored first, as plain Markdown. They specify behavior in a form that is mechanically checkable against source code directly: SQL WHERE predicates, Zod schema shapes, error codes and messages character-by-character, response shape, enforcement layers. The rule is the assertion. No intermediate test is required to express it.

Every domain file references seven shared authority files: _shared/auth.md, _shared/permissions.md, _shared/persistence.md, _shared/errors.md, _shared/audit-log.md, _shared/security.md, _shared/identifiers.md. These encode cross-cutting contracts — AUTH-5 (cross-org IDOR collapsed to a single NOT_FOUND), PER-2 (every SELECT includes notDeleted()), AL-6 (writeAuditLog never throws), SEC-2 (sanitizeText on all persisted free-text), ID-1 (zUuid on every UUID input). A cross-domain registry at .claude/docs/domain-rules/domain-registry.md tracks dependencies between domains.

Source code has zero authority. If the rules say X and source does Y, the source is wrong.

2. /contract extracts layers from rules

Before any code gets written, the /contract skill reads the domain rules file in full, all seven _shared/*.md authority files, and the domain registry. It then verifies every shared infrastructure dependency the rules reference against what is actually on disk. Grep for each utility’s function name in packages/api/lib/, find the real file, record the actual import path. Domain rules describe what utilities are needed, but they can have stale or incorrect file paths. The filesystem is truth.

For each procedure in the domain rules, /contract assigns a status: fix-existing (the file and procedure exist, source bugs are documented), build-new (no file or no procedure yet), or blocked-on (a shared utility the procedure requires is missing from disk).

Then it groups procedures into layers of 3 to 6 procedures each, using six grouping heuristics in priority order: shared consistency rules (the strongest signal, because procedures bound by three or more [consistency] rules must share a layer), same entity lifecycle (list/getById/create/update/remove on the same table), same permission gate, same sub-entity table, same external integration, same base procedure type (weakest). Layers are then ordered by dependency: reference data first, core CRUD second, sub-entity after parent CRUD, bulk after single-item, analytics after CRUD, AI integrations last.

The output is a contract document at .claude/docs/contract-sheets/{domain}.md with two sections: a Domain Preamble (40-80 lines — what the domain is, its primary tables, verified shared infrastructure with real import paths, permission model, cross-cutting rule IDs, layer summary) and one Layer Brief per layer (80-300 lines, containing procedures with VERBATIM spec rules, VERBATIM consistency rules, VERBATIM adversarial rules including the business rationale, _shared rule references, and infrastructure dependencies).

Adversarial rules must appear verbatim. An agent given the summary “validates valueCents bounds” will not handle valueCents=0 correctly. An agent given the full rule including “DR-40: valueCents=0 must be ACCEPTED for condemned property, donated property, or land under eminent domain” understands the reasoning and cannot rationalize against the rule. Summarizing destroys the reasoning.

For domains with seven or more procedures, /contract dispatches a read-only verification subagent with only the contract document and the domain rules in context. The subagent checks five criteria independently: procedure completeness, rule coverage, adversarial preservation (verbatim check), status accuracy, layer coherence.

The contract is marked EPHEMERAL in its header. Domain rules are the source of truth; the contract is a derived artifact regenerated on every /contract invocation.

3. /build-source implements one layer at a time

/build-source reads the contract document (Domain Preamble + the specific Layer Brief for the layer it’s implementing), the domain rules in full, all seven _shared/*.md files, and the source-references document. Then it reads the actual implementations of every shared utility the layer will call: packages/api/trpc/init.ts, packages/api/lib/audit.ts, packages/api/lib/soft-delete.ts, packages/api/lib/sanitize.ts, packages/api/lib/time.ts, packages/api/lib/zod-uuid.ts, packages/api/lib/pagination.ts. If any shared implementation contradicts its _shared/*.md contract, the skill stops and reports the drift. Shared bugs propagate into every dependent procedure; catching them here prevents the propagation.

Before writing any code, /build-source builds three pre-flight lookup tables that act as the verification oracle for every per-procedure check downstream:

• TABLE A — Error Messages. For every error every procedure throws: procedure, code, exact message character-by-character from the DR, rule reference.
• TABLE B — Audit Log Calls. For every mutation: procedure, action, entityType, product, metadata keys.
• TABLE C — ENTITY_TYPES pre-flight. For every entityType value in TABLE B, grep packages/api/lib/audit.ts for that literal string. If any entityType has zero matches in the ENTITY_TYPES array, the skill stops. The entityType has to be registered in both audit.ts and _shared/audit-log.md AL-3 before the procedure can be built.

Then for each procedure in the layer, a bounded loop runs:

1. Fill in the per-procedure pre-flight: base procedure type, Zod input fields using fixed templates chosen from a list rather than invented (e.g., zUuid, z.string().min(N).max(M).transform(sanitizeText), z.number().int().min(0).max(2147483647) for money), errors from TABLE A, audit log from TABLE B, adversarial rules verbatim from the Layer Brief.
2. Write the procedure, using the pre-flight as the specification.
3. Run 15 mechanical grep checks against the source file: input.propertyId appears outside the Zod schema (SEC-25), every .update( has notDeleted and org-scoping in its .where( within eight lines (TOCTOU), every z.string() for a persisted free-text field chains .transform(sanitizeText), every error message from TABLE A appears verbatim in the code, every audit log matches TABLE B, writeAuditLog is not wrapped in try/catch (AL-6), ?? "" is forbidden (AL-4 — use ?? null), console.error / console.log / console.warn are forbidden (use writeSystemErrorLog), legacy cursor validators (min(20).max(36)) forbidden, every .insert( / .update( has .returning() within ten lines, every SELECT has notDeleted in its WHERE, deletedAt does not appear in response shapes, ctx.user.id is forbidden (use ctx.session.userId per AUTH-8), new Date() is forbidden in routers (use getNow()), every FORBIDDEN error has a non-empty message.
4. Fix any failure before moving to the next procedure.

The Nth procedure gets the same 15 checks as the first. Fatigue-driven shortcuts on later procedures are the leading source of audit findings; the loop is rigidly symmetric by design.

After all procedures in the layer pass their individual checks, four cross-procedure consistency checks run: permission strings uniform against the canonical P-2 through P-10 messages, audit log product field uniform across all mutations in the domain, every .where( in a single-entity operation contains organizationId or propertyId within five lines, deletedAt / organizationId appear only in WHERE clauses or getTableColumns exclusions — never in returned objects.

Then gate:static runs as the hard gate: ESLint with --max-warnings 0, ast-grep against 41 constitutional patterns, dep-cruiser against the monorepo boundary rules, TypeScript strict typecheck. If the gate fails, the skill fixes violations and re-runs until clean.

Fix mode (--mode fix) applies the minimum change per documented bug; it does not rewrite working code outside the bug scope. If the contract has classified a bug as a “DOMAIN-RULE INCONSISTENCY” rather than a source bug, /build-source does not change source — it reports the finding for /specify edit to address the rule instead.

4. /review-source runs a 9-pass adversarial review

After /build-source completes all layers of a domain, /review-source reads the same authority documents the contract read, plus every router file for the domain, plus every shared infrastructure implementation the routers depend on. It then runs nine sequential passes over the code.

Pass 0 (Shared Infrastructure) uses the Extraction-Then-Verification protocol to prevent agent satisficing. It runs in three phases:

• Phase A (extraction). For each shared infrastructure file, produce a numbered table of every behavioral expression: every context access, every variable construction, every comparison, every function call, every default or fallback, every type constraint, every error handler, every I/O call. Line-by-line, raw extraction, no compliance evaluation.
• Phase B (rule-by-rule negation audit). For each rule in the corresponding _shared/*.md file, walk the extraction table twice: once listing COMPLIANT expressions, once listing VIOLATING expressions. Both columns are mandatory. A missing VIOLATING column means the check was incomplete, not that the code was clean. Citing entry numbers from Phase A prevents the reviewer from re-reading the file from memory.
• Phase C (mechanical grep sweep). Run the Grep tool against a forbidden-patterns registry (FP-1 through FP-19) covering patterns like ctx.user.id, new Date(), console.error, ?? "", bare FORBIDDEN errors. Grep is authoritative; if Phase C finds a match Phase A missed, the grep result wins.

Passes 1-8 walk the domain’s source procedure by procedure:

• Procedure Inventory. Every procedure in the domain rules exists in the router with the correct base.
• Input Validation. Every Zod field matches the DR field-by-field; free-text fields use sanitizeText, searches use escapeWildcards.
• Query Correctness. Every SELECT WHERE, every JOIN, every ORDER BY, every mutation .values() / .set() matches the DR; .returning() with empty guard on every mutation; getNow() and generateId(), never new Date() or crypto.randomUUID().
• Error Handling. Every error code and message matches the DR character-by-character; cross-org cases return NOT_FOUND per AUTH-5; error messages don’t interpolate user input per ERR-8.
• Audit Log Compliance. action, entityType, entityId, product, metadata all match what the DR specifies for THIS procedure (some DRs specify null metadata, some {}, some specific fields, so universal defaults are wrong).
• Response Shape Safety. deletedAt and organizationId excluded from returned objects; no raw ...row spreading; internal fields (sortOrder, internalNotes, smartyKey) don’t leak.
• Security Surface. IDOR checks in SQL WHERE (not post-fetch JS comparison); permission guards on every write; no sql.raw() in business logic; TOCTOU protected by transactions or unique constraints; error messages identical across “doesn’t exist” and “exists in another org.”
• Engineering Quality. Dead code, consistency, type safety (no as any), no empty catch blocks (except writeAuditLog per AL-6).

Every finding is classified twice. The actionClass drives remediation: FIX (source must change; the default, covering about 99% of findings), SPEC-GAP (rules must be ADDED — the source does something correct that the DR doesn’t document), or RISK (security or quality gap requiring action in the same cycle). The auditSeverity communicates investment risk for reporting: CRITICAL / HIGH / MEDIUM / LOW.

A hard rule governs SPEC-GAP: it can only ADD to domain rules, never remove, weaken, or delete. If the DR says the response includes field X but the source doesn’t return it, the source is wrong (FIX), not the DR (SPEC-GAP). Source code has zero authority; a SPEC-GAP that removes a requirement from the DR is itself a bug in the review.

The output includes a structured JSON block (<!-- REVIEWER_OUTPUT_START -->) that the control-plane reads mechanically. If clean: true is emitted with any FIX or RISK findings in findings[], the control-plane treats it as invalid and blocks the phase. A self-verification step before emitting the JSON enforces the contradiction check.

The review is read-only. It produces findings. It never modifies source.

5. QA auditor exercises real user journeys

This layer catches what static review cannot. The QA auditor is a bash orchestrator (scripts/qa-runner.sh), a Claude Code skill (.claude/skills/qa-audit/SKILL.md), and a set of YAML-defined journey flows under scripts/qa-flows/.

Orchestration. The runner spins up an isolated Next.js build on port 3011 so it never collides with the dev server on 3010. It seeds an auth state file via qa-seed-auth.ts that the Playwright MCP consumes as its --storage-state. It seeds cross-tenant fixtures via qa-seed-fixtures.ts for flows that need Org A and Org B data side by side. It loads every flow YAML via qa-load-flows.ts, then dispatches one claude -p subprocess per flow, each with Playwright MCP attached and the /qa-audit skill active.

One flow per dispatch. The skill enforces a strict contract: one invocation tests exactly one flow, produces exactly two reports (Markdown and JSON) at the paths the runner passed in, then stops. The STOP condition is enforced to the tool-call level: once both report files exist, zero additional tool calls are permitted. The bound prevents the drift mode where an agent “also checks the adjacent flow while it’s already logged in.”

Flow definition. A YAML file specifies name, group (auth, hub, po, cross, canary, security), route, viewports (default: desktop 1440x900 and mobile 375x812), and instructions. The instructions block states PASS CRITERIA, SEVERITY calibration for CRITICAL / HIGH / MEDIUM / LOW, and NOT-A-BUG exclusions. Optional fields: requires for ordering (po-tasks requires po-properties to run first), domain_rules for DR IDs that get inlined into the prompt so the auditor verifies specific invariants, console_allowlist regex patterns for infra noise that isn’t a finding, user_context (which auth state to bootstrap — a, b, or none), expected.*_findings_max per-severity caps.

What a single dispatch does. For each viewport in flow.viewports:

1. Resize the browser, navigate to the route, wait on a concrete DOM or URL indicator (never sleep), take a screenshot at an absolute path under screenshots_dir.
2. Capture browser_console_messages, apply the console_allowlist, report the survivors.
3. Capture browser_network_requests, any 4xx/5xx at an unallowed path is a finding.
4. Inject axe-core 4.11.x and run WCAG2A + WCAG2AA + best-practice. Each violation above moderate impact becomes a finding keyed a11y:<ruleId>. If axe fails to load, the flow is marked INCONCLUSIVE for a11y with no manual fallback.
5. Execute flow.instructions literally. Ambiguous steps are marked INCONCLUSIVE, not interpreted.
6. Verify persistence: after a create, reload the list, confirm the row is rendered. Navigate away and back, confirm again. A success toast is not sufficient.
7. Close the browser via browser_close so Chromium doesn’t orphan across flows.
8. Write both reports and exit.

Finding persistence. Per-run JSON reports get ingested into a SQLite database at qa-reports/_state/findings.db. Findings are deduped by a stable hash of (title, severity, flow), with columns for first_seen, last_seen, and runs_seen_in. A finding that appears once and disappears is flagged as flake. A finding that persists across runs ages in severity.

The regression gate. Per-flow expected.*_findings_max caps hard-fail the run when exceeded. Above the per-flow caps, a global regression gate compares the current run to the previous run with configurable slack (QA_SLACK_HIGH, QA_SLACK_MEDIUM, QA_SLACK_LOW). A new HIGH that wasn’t there yesterday blocks today.

The timezone sweep. qa-tz-sweep.sh runs the entire suite twice: once with TZ=America/Los_Angeles, once with TZ=Europe/London. Any flow whose findings differ between the two runs is a timezone bug. This is a class of defect tests almost never catch, because tests run in one timezone.

6. Regression tests written after QA

Once the QA auditor’s run on a domain is clean, a regression test is written for each bug it surfaced. Those tests use createCaller per the existing ast-grep rules, exercise production code, and cannot be silenced without editing the gate. The shape of a regression test is specific: reproduces the bug against real code, fails before the fix, passes after. Each test is tied to a finding ID in the SQLite database, so the test and the incident it prevents stay linked.

What each layer catches

• Rules catch what is mechanically checkable at the code level: SQL predicates, schema shapes, error codes, enforcement layers.
• /contract catches infrastructure mismatches (domain rules reference a utility at path X, real utility lives at path Y) and procedures that cannot be implemented yet (blocked-on a missing shared helper). It forces verification against the filesystem before any code is written against the assumptions.
• /build-source catches per-procedure violations at write-time via 15 grep checks plus three pre-flight lookup tables, then four cross-procedure consistency checks, then gate:static as the hard gate.
• /review-source catches cross-procedure drift and gaps that survive the per-procedure grep, including shared-infrastructure drift via the Pass 0 ETV protocol (extract behavioral expressions, rule-by-rule negation audit, forbidden-pattern grep sweep). It catches what satisficing hid.
• QA auditor catches what the running system does at the UI level: persistence across navigation, multi-viewport rendering, accessibility against rendered DOM, console noise, network errors, visual state transitions, timezone handling, prompt-injection hygiene. None of these can be asserted at the test-harness level.
• Regression tests lock behaviors that survived all five earlier layers against future edits.

Each layer catches a different class of failure. Removing any one layer leaves a class of bug nobody checks for.

What stays from the old apparatus

Tests stay. They move to the end of the pipeline. The ast-grep rules that block fake tests (require-caller-in-tests, require-router-import-in-tests) stay. The requirement that every test exercise production code via createCaller stays. What changes is the position: tests are no longer the specification. They are the lock on what the earlier layers already proved correct.

After the first round of domains under Source-First, the regression suite is approximately 200 tests. Each corresponds to a specific bug the QA auditor surfaced. Compared to the 10,000 tests an exhaustive test-first pipeline would demand, 200 is what is earned.