We built an invoice extractor that read a PDF, pulled the total and line items, and fed them into a costs rollup. It demoed clean. In production it broke roughly twice a week — a malformed number, a total that came back as a string, once a line item array that was null.
We logged it for a month and the model was wrong about 4% of the time. Not catastrophically, just enough that on 800 invoices a week, 30-odd went bad and nothing caught them. The fix wasn't a better prompt. It was treating the model like the flaky service it is.
Force a schema, then validate it
Our first version read prose back and pulled numbers out with a regex. That's where most of the 4% lived. We constrained the model to a schema and validated what came back, the way you'd validate any untrusted input at a boundary:
class Extraction(BaseModel):
invoice_total_cents: int
vendor_name: str
line_items: list[LineItem]
resp = call_model(prompt, response_schema=Extraction)
try:
result = Extraction.model_validate_json(resp)
except ValidationError as e:
result = retry_with_error(prompt, resp, e) # show the model its mistake
Structured output plus validation turned "usually roughly right" into "exactly this shape or the call fails loudly." When validation failed we retried with the error fed back in, and the model corrected on the retry far more often than it repeated the mistake — in our logs about 85% of retries came back valid. That's the retry loop you'd build around any flaky dependency, no different.
Two failure modes the schema didn't catch on its own, and both mattered. First, the schema said invoice_total_cents: int, but the model would sometimes return a valid integer that was wrong — it read the subtotal instead of the total. Type-valid, semantically wrong. We added a cross-field check: the line items had to sum to the total within a cent, and a mismatch triggered the retry with the discrepancy spelled out. That caught a class of errors a shape validator never will.
Second, the retry loop needs a cap or it becomes a cost bomb. We saw a malformed scanned invoice send the model into three, four, five retries, each one a full call, none converging. We capped at two retries and routed the third failure to a human review queue:
def extract(prompt, max_retries=2):
resp = call_model(prompt, response_schema=Extraction)
for attempt in range(max_retries + 1):
try:
result = Extraction.model_validate_json(resp)
if abs(sum(li.amount_cents for li in result.line_items)
- result.invoice_total_cents) > 1:
raise ValidationError("line items don't sum to total")
return result
except ValidationError as e:
if attempt == max_retries:
queue_for_human(prompt, resp, e) # stop retrying, escalate
raise
resp = retry_with_error(prompt, resp, e)
About 3% of invoices hit the human queue, which is roughly the 4% error rate minus what the retry recovered. Those go to a person instead of silently landing wrong in the rollup. A caught error you escalate is a completely different outcome from a wrong number nobody sees.
Verify adversarially
Separately we had a review-agent that read a diff and reported bugs. Asked to find a problem, it found one whether or not there was one. On a sample of 40 diffs it flagged 61 issues; when we checked them by hand, 23 were real and the rest were invented to satisfy the request. On its own it couldn't tell us which.
So we stopped letting the finding stand on the finder's word. We ran a second model whose only job was to refute it:
finding = model_find_bug(diff)
verdict = model_refute(finding, diff) # "try to prove this wrong"
if verdict.holds_up:
report(finding)
The refuter has a different prompt and no investment in the finding being real, so it killed most of the confident-but-wrong ones. It cut the 61 flags down to 26, of which 21 were the real ones — we lost 2 true positives and killed 35 false ones. That's the same instinct as not letting an author review their own PR. A single model grading its own work grades generously.
The two lost true positives are the honest cost, and we looked at them. Both were subtle — a real off-by-one the refuter argued its way out of because the surrounding code looked defensive. We decided a 2-in-23 miss rate on real bugs was an acceptable trade for cutting false positives from 38 to 5, because a review agent that cries wolf 38 times gets ignored, and an ignored reviewer catches zero. The math we cared about was signal-to-noise, not raw recall. Before the refuter, a developer reading 61 flags to find 23 real ones stops reading. After, 26 flags with 21 real is a list someone actually works through.
The prompt asymmetry is the mechanism, and it's worth being specific. The finder's prompt is "find bugs in this diff." The refuter's is "here is a claimed bug; construct the strongest argument that it is not a bug." Same model, opposite objectives. The refuter does more than double-check the finder — it actively tries to defeat the finding, and a claim that survives an adversary trying to defeat it is worth more than a claim the original author simply didn't doubt.
Determinism where the domain needs replay
The invoice classification fed a financial calculation, so it had to be reproducible. It couldn't return one answer today and a different one tomorrow for the same PDF, or the audit trail is fiction.
For those calls we pinned everything we could — temperature to 0, a fixed model version, prompt and inputs logged so the decision can be re-derived:
call_model(
prompt, temperature=0, model="pinned-version-id",
seed=record_id, # same input, same output
)
log_decision(record_id, prompt_hash, inputs, output)
You don't get bit-identical determinism from a model the way you do from a pure function. But this got us close enough to replay a decision and defend it, which was the actual requirement. When finance asked six months later why an invoice was classified the way it was, we had the prompt hash, the inputs, and the output on file.
The prompt hash is the part people skip, and it's the part that saved us. We hash the exact prompt template plus the model version and store it with every decision. Six months later the template had been edited twice. Without the hash, "why was this classified this way" is unanswerable, because you can't tell which version of the prompt produced the decision. With it, we could pull up the exact prompt text that was live on that date and show finance the reasoning was consistent with the rules in force at the time:
record = decisions.get(record_id)
prompt_text = prompt_versions.get(record.prompt_hash) # the exact prompt used
# now you can re-run or just show what was asked and answered
A decision you can't reconstruct is a decision you can't defend in an audit, and "the model said so" is not a defense. The temperature-0 pin gets you reproducibility on the model side; the prompt hash gets you reproducibility on the input side. You need both, because the prompt changes more often than the model does.
Agents that work versus agents that demo
The line between an agent that does real work and one that only demos is this list, not the model. The extractor used the same model before and after. What changed was the machinery around it.
A demo agent is a prompt and a loop. It looks magical on the happy path and falls apart on the 20th input, because nothing catches the malformed output, nothing retries the failure, nothing verifies the confident-wrong answer, and nothing is reproducible. That was ours in month one.
A working agent wraps the same model in the boring machinery you'd wrap any unreliable service in — schema at the boundary, validation, retry with the error fed back, an independent check on anything load-bearing, logs for replay. Treat the model like an oracle and you're betting production on it being right every time; it's wrong 4% of the time. Treat it like an API and that 4% becomes a caught error instead of a bad row in the rollup.