On the evening of March 30, 2026 I ran a 20-minute QA audit on software I had been building for six weeks. The audit was a sanity check before a client demo. I expected to find maybe two or three bugs.
The first failure hit within 90 seconds. A user in one organization could see properties from a different organization by typing the property UUID into the URL. IDOR, unambiguous, the kind of bug that ends a due-diligence call and stops a term sheet.
I opened the test file for that endpoint. The test suite passed. I opened the router. The procedure was missing a WHERE organizationId = ctx.session.activeOrganizationId predicate. Every test for that procedure passed because every test mocked the database and asserted that the mocked response matched itself.
The audit ran for the rest of that 20 minutes and kept finding bugs. Broken mutations that the tests said worked. Nil UUIDs going to production with test assertions confirming the “expected” UUIDs matched. Stored XSS in input fields that the tests validated the rendering of. Timezone corruption on every write, with tests asserting the wrong timezone was correctly applied. Dead routes that returned 200s and did nothing, backed by tests asserting the empty response was the expected empty response. Over 100 critical bugs total.
The test suite had 4,800 passing tests. None of them had caught any of it.
What a fake test looked like
I spent the afternoon tracing representative failures. Every test had roughly this shape:
vi.mock("./repository", () => ({
fetchProperty: () => ({ id: "abc", name: "Test Property" }),
}));
it("fetches a property", async () => {
const result = await getProperty({ id: "abc" });
expect(result.id).toBe("abc");
expect(result.name).toBe("Test Property");
});The mock returns a canned object. That object flows into the procedure, the procedure returns it, and the assertion compares it to itself. The production code path is never executed. If I deleted the entire repository file, the test would still pass.
This is what AI coding agents produce reliably when asked to write tests. The agent reads the procedure, sees a dependency, mocks the dependency with something plausible, and asserts against the mock’s return value. The resulting test is a tautology. It never fails because it cannot fail. It exercises its own setup.
The reason it passes human review is that everything visible in the file looks correct: imports are right, describe and it blocks are named reasonably, assertions use real matchers. Only a reader who traces the mock dependency back to production code notices that nothing real is being tested. Most reviewers do not do that trace, especially late at night, especially under delivery pressure.
Deleting everything
I spent that Friday night sitting at the test summary. 4,800 tests. I had written them. Or more accurately, I had directed a Claude Code session to write them, reviewed the file shapes, and shipped the pipeline when the suite turned green. The suite had been green for weeks. It had been the thing I pointed at when I wanted to feel like the project was in good shape.
I deleted all of them that weekend. The decision was not difficult once I understood what they actually proved. A test suite that passes with 100+ critical bugs in production is worse than no test suite: it generates confidence as its output, and the confidence is wrong in the direction that matters. Zero tests at least communicates that you have zero tests.
I told the client about it on Monday morning. He asked if this meant the project was delayed. I said yes.
The rules that closed the class
Over the next week I wrote two ast-grep rules that make the circular-mock pattern mechanically impossible to ship. Both run at commit time via pnpm gate.
id: require-caller-in-tests
language: TypeScript
rule:
all:
- pattern: it($DESC, $FN)
- not:
has:
any:
- pattern: createCaller($$$)
- pattern: createMockCaller($$$)
message: |
Every it() block must call createCaller(). A test that mocks
everything and asserts on the mock is a tautology.
See fake-tests-weekend incident.The first rule, require-caller-in-tests, scans every it() block and fails the commit if the block does not call createCaller(...) or its equivalent. createCaller is the tRPC helper that produces a real, ctx-bound caller against the actual router. A test that does not construct a caller is not executing production code, which means it is not testing production code.
The second rule, require-router-import-in-tests, fails the commit if a test file does not import a tRPC router at all. A test file with no router import cannot be exercising any procedure in that router. Every test file that mattered was importing a router it never called. Every test file that imported nothing was a tautology farm.
Neither rule can be silenced short of disabling the gate. I added a hookify rule on top of them that blocks --no-verify on the commit, so bypassing the gate requires editing git config, which another hook also blocks. Three layers, because verbal discipline had already proven insufficient in this exact area.
What the incident did to the methodology
I had been running Domain-Rules TDD at the time. Tests were written first from domain rules, then the code was written to pass the tests. The theory was that a test suite written from a specification could not drift from the specification. It turned out that the test suite could drift from reality without drifting from the specification, because the tests exercised mocked reality and the specification had nothing to say about whether the mock was accurate.
The 4,800 deleted tests were replaced over the next three weeks by about 1,800 tests that all had to call createCaller or they would not pass the gate. The new suite found 8 production bugs in its first week, bugs the old suite had been missing for months.
That realization did not land all at once. What landed first was the deletion. What landed over the next few weeks was a rebuild of how tests got written and what they ran against. The methodology that replaced Domain-Rules TDD is the one on this site: rules written first, code written directly from the rules, tests added only as regression locks after a live QA audit finds a bug worth locking.
The rules explorer has a rule that was born that weekend: every mock chain traces back to the real production router, or the test does not count.