Headline AI Numbers Hide Disagreement: Grader Refusal, Contested Labels and Lost Verdicts in AI Safety and Financial Evaluation

Decisions about artificial intelligence systems increasingly rest on single numbers, and each rests on three things that are seldom checked together: the human labels it is scored against, the AI grader that produces the verdicts, and the code that reads the grader's reply. This paper evaluates all three, which is to say it conducts a meta-evaluation that also checks the reference. On JailbreakBench, ten graders put the attack success rate anywhere between 9.8% and 49.0% against a human majority of 36.7%. Claude Opus 5.5 declines to grade 71% of its calls on the attempts humans judged successful against 20% on the rest, its refusal rate rises with each additional rater who called an item harmful on both a jailbreak and a violence dataset, and the pattern survives changes to the grader's framing; counting only what it grades places attack success at 7.6%. Where human raters themselves disagree, JailbreakBench's chosen judge and a shipped safety scorer do no better than a constant answer. Auditing sixteen handling rules across twelve evaluation tools and benchmarks, we find missing verdicts silently scored as safe, as failures, as maxima or dropped, and we set out a reporting standard, parts of which now ship in Microsoft's PyRIT. Joining the three, an uncertainty budget shows that the sampling error bar covers about a sixth of the full band around Opus's headline, with missing verdicts the largest share where a grader refuses often and the contested reference the largest where it seldom refuses, as on a third dataset in which 123 people rated each conversation. Sovereign credit ratings and Shariah screening show the same structure in finance. We therefore argue for a unified meta-evaluation, one that treats the human reference, the grader and the code that reads it as a single measurement problem and reports one uncertainty for the headline in place of three separate caveats.