Two Blind Judges, One Arbiter: A Reliability Audit of LLM Labeling Decisions in a Production Pipeline

A methods report auditing a production pipeline that labels short financial news with three-class sentiment using two blind LLM judges from different model families (Claude Opus 5, GPT-5.6) and a third model (Claude Fable 5.1) as arbiter. It analyzes 34,998 judged rows, a 1,250-item news-to-market matching task, and the frozen-test errors of a FinBERT classifier trained on the final labels, with packet-level bootstrap intervals. Findings: disagreement is a neutral-boundary threshold problem (94% of disputes); stated confidence predicts disputes (AUC 0.84) but not correctness (0.56); re-arbitrating 595 disputes under eight conditions ties the arbiter's 70% lean toward one judge to the production session itself rather than to model family or to the information shown (information explains 3 points); fresh arbiters of every family default to neutral 82-96% of the time; classifier error is 3.4% on confidently agreed rows and 42.9% on arbitrated rows, and 82% of those errors equal the losing judge's label. Reports in English and Turkish, analysis code, and row-level tables are included.