Why Escalate? Typed Escalation for LLM-Judge Panels with Per-Type Conformal Guarantees

LLM judges are increasingly deployed with a reject option. Recent methods certify the error rate of the verdicts these judges accept, but items that are escalated carry no reason, even though conflicting evidence, missing evidence and a false premise call for different remedies. We ask whether a panel of LLM judges can say why it escalates, with a separate statistical guarantee for each reason. Learn-then-Test acceptance is kept, and each escalated item receives a set of escalation types calibrated by per-type split conformal prediction. For each type separately, an escalated item that carries the type includes it with probability at least 0.9 (marginally, for the constructed labels, assuming exchangeable items); for the whole set, only the weaker union bound holds. Typed items were built on SQuAD 2.0 without annotators (conflict, scarcity, both, false premise), and the panel had three judges from different model families (4,801 items). In the preregistered analysis (protocol amended after a pilot that selected the scores), mean per-type coverage over 200 article-level re-partitions was 93.8% (conflict), 91.2% (scarcity) and 94.0% (false premise), and accepted verdicts had a selective risk of 7.5%. Because items are clustered in articles, we added an exploratory article-level variant that is exactly valid when articles are exchangeable. Per-type coverage stayed above 0.9, but sets grew from 1.74 to 1.91 types, and exact article-level acceptance could not be certified with 74 calibration articles. Typed sets contained every true reason for 91.2% of typed escalations; that is higher coverage of reasons, not a measured operational gain. In further exploratory analyses, counting accepted items as misses, end-to-end detection was 42.1%, 69.2% and 35.9%, and on an external false-premise set it was 4.0% and stayed low after recalibration. A supervised score on the same panel outputs gave smaller sets. In an executed retrieval pilot, retrieval resolved most escalations that truly lacked evidence but few others; because the flags put scarcity in most sets, typed routing saved little over retrieving for everything. An automated check by two LLMs flagged 12.9% to 28.5% of conflict labels as doubtful; a human audit is pending. The glut of the existential triple (Dempster’s conflict) was far worse at detecting conflict than asking the judges directly.