Six Nameplates, Two Effective Votes: A Pre-Registered Local Replication of Correlated LLM Judge Errors, with a Deterministic Instrument for Measuring Panel Independence

Kohli (2026, arXiv:2605.29800) showed that nine frontier LLM judges deliver only ~2.2 effective independent votes because their errors are correlated, and that the correlation is mostly not explained by item difficulty. This deposit contains a pre-registered replication on consumer hardware (six local open-weight judges, 8B to 35B parameters, four base models, 1,000 ChaosNLI-MNLI items) and a deterministic, bit-replayable instrument that converts judge logs into per-pair corroboration weights via a cross-fitted Generalised Covariance Measure over pre-registered raw-input feature extractors. All four pre-registered predictions were confirmed: effective votes 2.13 for k=6 (Kohli: 2.18 for k=9); two judge pairs known by construction to share base weights ranked first and second in error correlation, one producing identical labels on 1,000/1,000 items and receiving corroboration weight exactly 0; and pre-registered raw-text difficulty features explained approximately none of the mean pairwise error correlation. On the Condorcet-gap estimand, this deposit's documented implementation of Kohli's framework attributes 27.8 percent of the panel's gap to difficulty at ten entropy bins (sensitivity 22.1 to 36.4 across variants) against his 13.5, with the majority of the gap surviving difficulty conditioning in both studies; an earlier cross-estimand comparison in v1.0.0 was withdrawn, and the erratum at the top of the note carries the correction. v1.2.0 adds a registered self-replication. The deposit's diagnostic, reimplemented as the full corroboration layer of the framework named below and run over this deposit's own judge logs under five pre-registered predictions, reproduces the headline at n_eff = 2.1311 under a different estimator (computed on the 997-item complete-case intersection; the deposit's 2.13 is computed on the full 1,000 with three unparsable outputs scored as errors), again prices the identical pair at exactly zero, again ranks the shared-base pairs lowest on measurement alone and prices them at zero by attestation under provenance fingerprints, and reproduces the surface-form null (conditioning delta 0.0015). The registered record and full results are included under replication/; the layer implementation belongs to a wider unpublished programme and is not part of this deposit. The deposit includes the technical note, the pre-registration (content hashes fixed before any judge ran), the complete instrument (pure Python standard library, no RNG), all 6,000 raw judgments with raw completions preserved, and the content-hashed analysis report. The ChaosNLI dataset is not redistributed; the deposit README explains how to regenerate the exact sample and verify its hash. The instrument is a standalone application of the measured-corroboration layer of the OFM-PBL / (In)Canon framework (UK Patent GB2606072.3): validator independence is never asserted, only measured, discounted by shared provenance, and capped below certainty.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-01
DOI
https://doi.org/10.5281/zenodo.21737025
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Six Nameplates, Two Effective Votes: A Pre-Registered Local Replication of Correlated LLM Judge Errors, with a Deterministic Instrument for Measuring Panel Independence

David Antonio Tomé
Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
preprint

Six Nameplates, Two Effective Votes: A Pre-Registered Local Replication of Correlated LLM Judge Errors, with a Deterministic Instrument for Measuring Panel Independence

David Antonio Tomé
preprint en

Abstract

Kohli (2026, arXiv:2605.29800) showed that nine frontier LLM judges deliver only ~2.2 effective independent votes because their errors are correlated, and that the correlation is mostly not explained by item difficulty. This deposit contains a pre-registered replication on consumer hardware (six local open-weight judges, 8B to 35B parameters, four base models, 1,000 ChaosNLI-MNLI items) and a deterministic, bit-replayable instrument that converts judge logs into per-pair corroboration weights via a cross-fitted Generalised Covariance Measure over pre-registered raw-input feature extractors. All four pre-registered predictions were confirmed: effective votes 2.13 for k=6 (Kohli: 2.18 for k=9); two judge pairs known by construction to share base weights ranked first and second in error correlation, one producing identical labels on 1,000/1,000 items and receiving corroboration weight exactly 0; and pre-registered raw-text difficulty features explained approximately none of the mean pairwise error correlation. On the Condorcet-gap estimand, this deposit's documented implementation of Kohli's framework attributes 27.8 percent of the panel's gap to difficulty at ten entropy bins (sensitivity 22.1 to 36.4 across variants) against his 13.5, with the majority of the gap surviving difficulty conditioning in both studies; an earlier cross-estimand comparison in v1.0.0 was withdrawn, and the erratum at the top of the note carries the correction. v1.2.0 adds a registered self-replication. The deposit's diagnostic, reimplemented as the full corroboration layer of the framework named below and run over this deposit's own judge logs under five pre-registered predictions, reproduces the headline at n_eff = 2.1311 under a different estimator (computed on the 997-item complete-case intersection; the deposit's 2.13 is computed on the full 1,000 with three unparsable outputs scored as errors), again prices the identical pair at exactly zero, again ranks the shared-base pairs lowest on measurement alone and prices them at zero by attestation under provenance fingerprints, and reproduces the surface-form null (conditioning delta 0.0015). The registered record and full results are included under replication/; the layer implementation belongs to a wider unpublished programme and is not part of this deposit. The deposit includes the technical note, the pre-registration (content hashes fixed before any judge ran), the complete instrument (pure Python standard library, no RNG), all 6,000 raw judgments with raw completions preserved, and the content-hashed analysis report. The ChaosNLI dataset is not redistributed; the deposit README explains how to regenerate the exact sample and verify its hash. The instrument is a standalone application of the measured-corroboration layer of the OFM-PBL / (In)Canon framework (UK Patent GB2606072.3): validator independence is never asserted, only measured, discounted by shared provenance, and capped below certainty.

Zenodo (CERN European Organization for Nuclear Research)
University of Stirling (GB), Plastic Logic (United Kingdom) (GB)
Peace, Justice and strong institutions
Explainable Artificial Intelligence (XAI)
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.