A wrong floor turns working alarms into background noise.

In a production filing system I run, the documented test-suite floor said twelve tests were known-fail, plus one that depended on the environment. An audit counted eighteen. Every one of the eighteen was a real defect in the tests or the code — none were environmental. Five of the mislabelled tests guarded the payment webhook that decides whether a customer’s payment is recognised, and they had been dead for months while the label said “expected.” This piece is about the discipline that finding forced — because “evals” is not a tool you install. It is a set of habits about what counts as evidence.

The accepted-failure list is a loan

Every team has one, formally or informally: the tests everyone knows are red, the flaky ones, the “that one needs the database.” The list feels like pragmatism. It is actually a loan against your alarm system, and the interest compounds invisibly: every mislabelled failure teaches the team that red is normal. Once red is normal, a new red — the one that matters — arrives pre-dismissed.

The fix was not heroic. Each of the eighteen was either repaired or its defect promoted to a tracked, executable reproduction. Five of them turned out to share a single root cause: a migration whose downgrade() was pass — the revision marker moved while the schema didn’t, which had already stranded a development database beyond the tooling’s reach. The suite now holds a zero-failure floor on both of its lanes, both lanes block deployment, and the floor lives in one machine-checked file — because the version that lived in prose was repeated in four documents and had drifted in three of them.

What eval discipline actually consists of

Expected values come from the source, before the code. For anything that computes a number someone signs, the expected value is derived by hand from the regulation and written into the test first. The one rule that must never bend: when a test fails, you do not update the expected value to match what the code produced. That proves nothing, and it bakes an unverified figure into the record as if it were truth.

Golden masters characterise; they do not bless. A characterisation harness records what the system does across a matrix of shapes and diffs every change against it. Its baseline may contain values that are known to be wrong — that is fine, because the value is the diff, not the content. The operating rule: when it fails, read the diff; never re-record to make it green. Every changed field must be one the change intended. Two details decide whether such a harness stays useful or rots: hunt down the volatile fields early (run it twice on unmodified code and see what “changes”), and choose the matrix to cover the shapes production does not currently run — a golden master over only your live traffic reproduces your live blind spots.

Defects live as strict expected-failure reproductions, not register prose. A bug that matters gets a test that reproduces it, marked expected-fail in strict mode. The reproduction and the regression guard are the same assertion. Strict mode means the day the defect is accidentally fixed, the suite fails loudly until someone removes the marker — so a defect can neither silently persist nor silently heal. Prose registers drift from the code; an executable reproduction cannot.

Coverage grows by state, not by count. Three confirmed money-handling defects in that system survived thousands of passing tests, because each needed a combination of conditions no fixture ever built — the ownership mode and the real input shape and an assertion on the value; the ledger debit and two concurrent sessions. A test that never constructs the state cannot catch the bug. When you add tests, vary the state space, not the call count.

LLM stages get replay pins. The model call itself is nondeterministic, but everything around it — routing, preprocessing, tool-call parsing, output validation — is ordinary code. Record a real model response once, then replay it through the full production path in CI: the deterministic ninety-five percent of the pipeline is regression-pinned at zero cost, zero network, zero flakiness. Re-recording is a conscious re-pin, exactly like re-recording a golden file.

The floor rule

Whatever your suite’s current honest state is, write it down as a floor in one machine-checked place, gate the pipeline on it, and adopt the only sane update policy: the floor only moves up. If the floor is currently “three known failures,” fine — say so, in a file the CI reads, with each failure named. What killed the alarm system in my case was not having failures; it was the gap between the documented number and the true one. The gap is where trust dies.

Five questions to ask your own suite

1 · Does an accepted-failure list exist — in a file, a wiki, or the team’s heads? When did someone last re-derive it from scratch?
2 · Can a defect in your system be fixed without any test noticing — or persist without any test failing?
3 · When a test disagrees with the code, which one wins by default — and who checks against the source before the expected value changes?
4 · What fraction of your LLM pipeline is pinned by replayed responses rather than live calls or hope?
5 · If your CI is green today, what precisely does that certify — and is the certificate written anywhere a machine checks?

The reason to care goes beyond hygiene. As models write more of the code, the scarce skill shifts to exactly this: being able to say what “correct” means, prove it held, and notice when it stops holding. That is an eval discipline — and it is worth naming as one.

← All insights