Ratio checks can’t catch a two-row drop.

A language model extracted a financial table, returned 52 of 54 rows, and silently dropped the last two. Every plausibility check passed. This is the story of the most dangerous kind of bug an LLM pipeline can have — and why the fix was a set difference, not a better prompt.

The bug that looked like nothing

The pipeline ingested trial balances — accounting tables, often photographed or exported in whatever format the client’s software produced. For large documents, extraction ran as a chunked fan-out: the model read sections in parallel, and the results were merged into one set of accounts.

One day, a merge came back two rows short. Not empty. Not garbled. Not an error. Fifty-two clean, correctly-typed rows out of fifty-four — and the two missing ones were simply not there.

The pipeline had checks. Totals were compared against expectations: in range. Ratios between account categories: plausible. A completeness floor — the extraction had to return at least 80% of the estimated rows: sailed over it without a murmur. The result was persisted, and the run reported success.

Why every check passed

All of those checks share one property: they measure plausibility, not completeness. They ask “does this output look like the kind of thing we expect?” — and a ledger missing two accounts looks almost exactly like a ledger that never had them. There is no statistical signature of absence. The totals shift by a fraction, well inside the noise any real dataset carries.

This is worth stating as a general law, because it applies to every extraction pipeline, not just financial ones: statistical checks can only catch statistically visible failures. A dropped row, a skipped invoice line, one product missing from a catalogue of hundreds — these are individually invisible to any threshold you can responsibly set. Set the floor at 99% and you reject good runs on noisy documents. Set it at 80% and a two-row drop passes forever.

And in a regulated system, a silent two-row drop is the worst possible failure. A crash is free: someone re-runs the job. A wrong number that looks right computes cleanly, passes review, renders a professional PDF — and surfaces eighteen months later as an assessment with penalties attached.

The fix was not a prompt

The instinctive response is prompt engineering: tell the model, more firmly, to return every row. That treats a structural gap as a discipline problem. The model already “knew” it should return every row. It usually does. Usually is the whole issue.

The actual fix has three parts, and none of them touches the prompt:

The set difference is the load-bearing part. Identity checks are binary where ratio checks are fuzzy: a row is either present or it is not, and no amount of plausibility can talk its way past that. The deterministic parse doesn’t need to understand the rows — amounts, meanings, mappings stay the model’s job. It only needs to know which rows exist. That is a much easier problem, and code solves it exactly.

The rule this became

The incident hardened into a rule that now applies to every model output in the system: model output is reconciled against something deterministic before it becomes data. Not spot-checked. Not sampled. Reconciled — against a source the code parsed itself, by identity, row by row.

The same pattern generalizes wherever a model transforms structured input. Injecting pre-composed identifiers into the model’s context — so it copies rather than composes them — turns the injected set into a completeness contract you can check mechanically. Arithmetic identities (closing = opening + movement, within a tolerance) catch corruption the same way set differences catch loss.

What to ask of your own pipeline

If you run any extraction pipeline with a model in it, three questions:

1 · If the model silently dropped 4% of the rows tomorrow, would anything fire? Not “would totals look off” — would something fire?
2 · Does any deterministic parse of the source exist, independent of the model, that knows what the output must contain?
3 · When reconciliation fails, does the pipeline repair surgically and then fail loudly — or does it retry the whole thing and hope?

If the answer to the first question is no, you don’t have a validation problem. You have a wrong-number-that-looks-right problem, and you won’t know until it costs something.

← All insights