Six Nameplates, Two Effective Votes: A Pre-Registered Local Replication of Correlated LLM Judge Errors, with a Deterministic Instrument for Measuring Panel Independence
Kohli (2026, arXiv:2605.29800) showed that nine frontier LLM judges deliver only ~2.2 effective independent votes because their errors are correlated, and that the correlation is mostly not explained by item difficulty. This deposit contains a pre-registered replication on consumer hardware (six local open-weight judges, 8B to 35B parameters, four base models, 1,000 ChaosNLI-MNLI items) and a deterministic, bit-replayable instrument that converts judge logs into per-pair corroboration weights via a cross-fitted Generalised Covariance Measure over pre-registered raw-input feature extractors. All four pre-registered predictions were confirmed: effective votes 2.13 for k=6 (Kohli: 2.18 for k=9); two judge pairs known by construction to share base weights ranked first and second in error correlation, one producing identical labels on 1,000/1,000 items and receiving corroboration weight exactly 0; and pre-registered raw-text difficulty features explained approximately none of the mean pairwise error correlation. On the Condorcet-gap estimand, this deposit's documented implementation of Kohli's framework attributes 27.8 percent of the panel's gap to difficulty at ten entropy bins (sensitivity 22.1 to 36.4 across variants) against his 13.5, with the majority of the gap surviving difficulty conditioning in both studies; an earlier cross-estimand comparison in v1.0.0 was withdrawn, and the erratum at the top of the note carries the correction. v1.2.0 adds a registered self-replication. The deposit's diagnostic, reimplemented as the full corroboration layer of the framework named below and run over this deposit's own judge logs under five pre-registered predictions, reproduces the headline at n_eff = 2.1311 under a different estimator (computed on the 997-item complete-case intersection; the deposit's 2.13 is computed on the full 1,000 with three unparsable outputs scored as errors), again prices the identical pair at exactly zero, again ranks the shared-base pairs lowest on measurement alone and prices them at zero by attestation under provenance fingerprints, and reproduces the surface-form null (conditioning delta 0.0015). The registered record and full results are included under replication/; the layer implementation belongs to a wider unpublished programme and is not part of this deposit. The deposit includes the technical note, the pre-registration (content hashes fixed before any judge ran), the complete instrument (pure Python standard library, no RNG), all 6,000 raw judgments with raw completions preserved, and the content-hashed analysis report. The ChaosNLI dataset is not redistributed; the deposit README explains how to regenerate the exact sample and verify its hash. The instrument is a standalone application of the measured-corroboration layer of the OFM-PBL / (In)Canon framework (UK Patent GB2606072.3): validator independence is never asserted, only measured, discounted by shared provenance, and capped below certainty.
Authors
- David Antonio Tomé
Institutions
- University of Stirling (GB)
- Plastic Logic (United Kingdom) (GB)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-09-01
- DOI
- https://doi.org/10.5281/zenodo.21737025
- Primary Topic
- Explainable Artificial Intelligence (XAI)
- Type
- preprint