MEASURING VERIFIER LEAKAGE FOR LLM JUDGES: STRICT BREACH RATES ARE A LOWER BOUND ON PAYMENT ERRORS

How do you measure how much fabricated work an LLM judge lets through, when an unsure judge can hedge rather than reject outright? The standard instrument is an automated strict breach rate (the fraction a judge accepts that an oracle panel unanimously rejects), and our central result is a measurement caution for any LLM verifier: a strict breach rate is only a lower bound on a judge's real leak (a leak being a fabrication the verifier wrongly accepts). The motivating application is a paid autonomous agent, where a confident fabrication draws the money before any after-the-fact rating can catch it and a settled coin does not return, so the verifier's leak is a payment error. Most fabrications land in a contested zone the strict rule never counts: the in-loop oracle auto-pays but hedges to "partial" rather than rejecting. Re-adjudicating that zone with an independent multi-vendor panel turns 23 strict-counted leaks into 104 confirmed across five judges (a ~4.5× undercount), and even the frontier judge's "0 strict breaches" hid 4. The direction is forced (a strict rate that skips hedged auto-pays can only under-count), so the open question is whether the skipped region is large; here it is, and the gap replicates in a second domain (23 real scientific abstracts, strict 3 → 17 audit-confirmed, ~5.7×), so it is not a contracts artifact. Two further results scope and frame the measurement. A payment-decision lens (scoring verification by the accept/withhold decision rather than a label) exposes a model-free span-ablation gate as a regression for single-document contracts. And a capability gradient on a 40-document corpus of real SEC-filing (EDGAR) contracts under a 12-round adaptive attack shows judge strength, not vendor, separates leaky from robust judges, a contrast that holds within a single vendor and across vendors (breaches cross-vendor by construction). A calibrated seeded-liar benchmark instruments the study.