Calibrating LLM-as-Judge Breach Detectors, and Auditing Them Against Their Own Labels

Every breach verdict in an LLM red-team is itself an LLM judgment, so the judge is the load-bearing component. It is also generic: “did the agent breach policy?” has the same shape for harm, data disclosure, or an unauthorized action. We calibrate breach judges across these classes from one template, a consummation gate: engagement is not a breach, consummation is. And we hold the whole pipeline to one rule, the independence discipline: ground truth is never the regulation text, the operator’s own decisions, or the verifier’s own score. That rule is the paper’s spine, and it indicts us first. We contribute three things. First, a transferable method: the per-type gate paired with a self-diagnosing harness that ships or refuses each judge on a measured threshold rather than rubber-stamping a prompt. Second, the independence discipline, applied to our own labels as strictly as to anyone else’s. Third, a measured finding: the security-relevant breach definition and the one a standard execution-grounded benchmark encodes diverge on one real trajectory in seven. The gate itself is prior work’s; these three are what we add. The least-contestable evidence is external. On JailbreakBench’s human-majority labels the gate moves a judge from last of five public classifiers to the frontier (70.3 → 91.0%, level with the strongest at 90.3–90.7%; we read the sub-point margin as parity, not a rank-1 claim). But three of the four classes are scored against labels we authored, which the discipline forbids. So we apply it to ourselves: an independent six-labeler panel relabels our own ground truth. On our most-promoted headline, a single-annotator evidence-modality κ effect, it did not replicate at this scale (raw ∆κ = +0.011). The panel diverged from our labels in both directions (45 verdicts, released case by case) without ever forming a majority against them. Re-instantiated across the three designed classes the gate calibrates at 97.3/98.9/96.9% agreement (descriptive, against operator labels, not external ground truth), and the harness earns its keep by self-diagnosing REFINE on the unauthorized-action judge before a cycle ships it. A two-family transfer test shows the gate’s recall is model-portable across judge families while its over-flag suppression is not. An external anchor (AgentDojo’s environment-state labels, n=144) confirms the split: recall holds at 100% against state-transition truth, while the residual agreement gap (21/144, one-sided, released for the reader to adjudicate) is a definitional difference between two objective labels, not over-flagging. That wedge is the finding: an “externally validated” breach-judge number can certify the narrower construct rather than the one that matters.