Cut short: output limits remove an LLM judge's rejections first

Evaluations that use a large language model as a judge usually cap the judge’s output length. A reply truncated before its verdict is then excluded from scoring or counted as a failure. Excluding them is unbiased if verdicts are missing completely at random, and counting them as failures is biased whenever any are lost. Diagnostic-accuracy research showed in the 1980s that excluding unreadable test results biases accuracy when their cause is related to the disease or to the result itself; LLM-judge evaluation has not yet borrowed that lesson. I came across a case where the missing verdicts did not look random, and tested the question in two pre-registered studies. Each study used four judges: 300 claims from HoVer (fact verification) and 300 solutions from ProcessBench (maths step checking). In all eight judge-and-task pairs, rejections were longer than acceptances, so a limit that cuts into the bulk of replies truncates rejections first. At a limit near each judge’s median reply length (on maths, the median of a separate calibration set), 56 to 90% of rejections were truncated, against 26 to 42% of acceptances (registered on maths, exploratory on claims). Excluding truncated replies therefore made every judge look more lenient (exploratory). The false-rejection rate fell from 32–45% to 12–29% on claims and from 3–7% to 0–1% on maths, and the prevalence of correct solutions in the scored maths set rose from 50% to 69–83%. On claims, where errors also ran long, accuracy rose by 7 to 9 points at the median limit (exploratory). The registered test, excluding each judge’s longest 30% of replies, gave about 5 points for the two direct judges and 2 to 3 points, not clearly above zero, for the two reasoning judges. Several predictions failed. On maths, errors ran clearly longer for only one judge, and a limit of twice that length was not shown to be safe for any judge; released replies from 11 judges on a pairwise benchmark, including GPT-4o, Claude 3.5 Sonnet and o1-preview, showed no clear effect. Two widely used evaluation tools ship default limits of 256 and 300 tokens, and 2 of 30 randomly sampled ACL Anthology papers report their judge’s limit. Judges should run with a limit they never reach, and evaluations should report missing verdicts by class, as the STARD guideline already asks of medical test studies. The most important untested case is LLM judges of clinical AI, such as AI scribes, which a third study, registered in October 2026, will test.