Punished for Being Right: Version Leakage in LLM-Judged Evaluation of Peer Review

Studies that compare the accuracy of AI and human peer reviews increasingly check review claims against papers posted on OpenReview. For ICLR 2024 the public version is the one revised after the reviews, so a reviewer who correctly flagged a problem that the authors later fixed is scored as wrong. We measure the size of this version leakage. Four LLM judges from four vendors check the claims of human and AI reviews of 50 ICLR 2024 papers. Against the public version, all four find AI claims contradicted less often than human claims, at about one fifth of the human rate under their majority vote. On 17 papers with a pre-review arXiv version, the public version overstates the human error rate by 50%, and the AI–human gap narrows by +0.037 [+0.014, +0.058]. The AI-to-human error ratio moves from 0.20 to 0.32, a change whose interval includes zero. A single judge on 62 papers finds an overstatement of 35% (45% on the 45 papers outside the four-judge subset). The penalty falls disproportionately on the comments that authors acted on. On the four-judge papers, human claims that the authors conceded and addressed in a revision are 10% of human claims but account for 42% of the net decrease in human errors on the pre-review version. AI claims remain less often contradicted on the pre-review versions, and the remaining gap is concentrated in checklist and clarity claims.