SEXTANT: Measuring Six-Nines Residual Error for LLM Decisions Behind Independent Deterministic Checks

No language model is accurate to six nines, yet process automation is often described with such figures. We study a narrower quantity that can be measured: the residual error of one model step wrapped in deterministic checks, defined as the probability that an output is wrong and accepted. On synthetic payment documents with truth known by construction, one model (Claude Haiku 5.5) was wrong on about 9% of documents. We report four results. First, the cheap way to certify a low residual, multiplying the model's error rate by the checks' miss rate on injected errors, failed: it bounded the residual at 9.19 × 10−4 while a direct count on 300,000 documents found 4.91 × 10−3, 5.3 times higher. The model's errors were not like injected errors. Every wrong output that passed the checks was of one kind, located exactly where the checks could not see. Second, one further deterministic rule closed that gap, and a direct zero-failure test on fresh documents found no wrong accepted output in 300,000 and then in 3,000,000 documents, a one-sided 95% upper bound of 9.99 × 10−7 per output processed for the 2,000 vendors of the test world. Because errors cluster by vendor, that bound does not carry over to new vendors (about 2 × 10−5 for a new set). A repeat with the same model through the provider's interface, on a fresh world of 50,000 vendors with about 60 documents each again found no wrong accepted output in 3,000,000 documents (bound 9.99 × 10−7), and 1.78 × 10−6 for a new set of vendors from the same generator. The price was automation: the share of outputs accepted without review fell from 91.6% to 69.7% on the 2,000-vendor world. Third, the same checks ported to a FlowX integration workflow, run on the FlowX platform in a demo (non-production) environment with a second model (GPT-6 Luna), gave verdicts identical to the reference implementation on every one of 2,999,805 documents run end to end, and accepted 0 wrong outputs among them, a one-sided 95% bound of 9.99 × 10−7 per output processed on the 2,000-vendor world. A second pre-registered platform count, on the 50,000-vendor world, accepted no wrong output in 599,952 documents, a bound of 5.58 × 10−6 for a new set of vendors: five nines, not six. Outputs the checks could not verify were not accepted; the use case decides whether such an output is rejected, retried or sent to a person. They were the model's errors (about 8%) and correct outputs the ledger could not tell apart from another invoice (about 22% on this deliberately hard synthetic mix). Fourth, on 387 real invoices most of the model's errors (69%) were of a kind injected errors never produce. The bounds below 10−4 concern one synthetic task. They support no statement about production systems or real invoices, and the real-invoice results are percent-level. What they do support is a method and a warning: a residual error rate is a property of a model, a prompt, a set of checks and a document mix together, and it has to be counted, not composed. Laboratory study on synthetic documents. Not a production measurement.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24339367
Primary Topic
Adversarial Robustness in Machine Learning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

SEXTANT: Measuring Six-Nines Residual Error for LLM Decisions Behind Independent Deterministic Checks

Bogdan Raduta
3 links
Zenodo (CERN European Organization for Nuclear Research)
Adversarial Robustness in Machine Learning
preprint

SEXTANT: Measuring Six-Nines Residual Error for LLM Decisions Behind Independent Deterministic Checks

Bogdan Raduta
preprint en

Abstract

No language model is accurate to six nines, yet process automation is often described with such figures. We study a narrower quantity that can be measured: the residual error of one model step wrapped in deterministic checks, defined as the probability that an output is wrong and accepted. On synthetic payment documents with truth known by construction, one model (Claude Haiku 5.5) was wrong on about 9% of documents. We report four results. First, the cheap way to certify a low residual, multiplying the model's error rate by the checks' miss rate on injected errors, failed: it bounded the residual at 9.19 × 10−4 while a direct count on 300,000 documents found 4.91 × 10−3, 5.3 times higher. The model's errors were not like injected errors. Every wrong output that passed the checks was of one kind, located exactly where the checks could not see. Second, one further deterministic rule closed that gap, and a direct zero-failure test on fresh documents found no wrong accepted output in 300,000 and then in 3,000,000 documents, a one-sided 95% upper bound of 9.99 × 10−7 per output processed for the 2,000 vendors of the test world. Because errors cluster by vendor, that bound does not carry over to new vendors (about 2 × 10−5 for a new set). A repeat with the same model through the provider's interface, on a fresh world of 50,000 vendors with about 60 documents each again found no wrong accepted output in 3,000,000 documents (bound 9.99 × 10−7), and 1.78 × 10−6 for a new set of vendors from the same generator. The price was automation: the share of outputs accepted without review fell from 91.6% to 69.7% on the 2,000-vendor world. Third, the same checks ported to a FlowX integration workflow, run on the FlowX platform in a demo (non-production) environment with a second model (GPT-6 Luna), gave verdicts identical to the reference implementation on every one of 2,999,805 documents run end to end, and accepted 0 wrong outputs among them, a one-sided 95% bound of 9.99 × 10−7 per output processed on the 2,000-vendor world. A second pre-registered platform count, on the 50,000-vendor world, accepted no wrong output in 599,952 documents, a bound of 5.58 × 10−6 for a new set of vendors: five nines, not six. Outputs the checks could not verify were not accepted; the use case decides whether such an output is rejected, retried or sent to a person. They were the model's errors (about 8%) and correct outputs the ledger could not tell apart from another invoice (about 22% on this deliberately hard synthetic mix). Fourth, on 387 real invoices most of the model's errors (69%) were of a kind injected errors never produce. The bounds below 10−4 concern one synthetic task. They support no statement about production systems or real invoices, and the real-invoice results are percent-level. What they do support is a method and a warning: a residual error rate is a property of a model, a prompt, a set of checks and a document mix together, and it has to be counted, not composed. Laboratory study on synthetic documents. Not a production measurement.

Zenodo (CERN European Organization for Nuclear Research)
Adversarial Robustness in Machine Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon