When the Judge Becomes the Experiment

Large language models are increasingly used as evaluators, but a stable candidate output does not guarantee a stable measurement profile. I reconstruct several evaluator interventions over frozen outputs from one high-stakes decision-support evaluation program and treat the evaluator itself as the changing measurement object. Four completed managed-evaluator runs over the same 600 cases produced 2,400 judgments. Repeatability was high: 595 of 600 cases were four-of-four stable; Fleiss' kappa was 0.993 overall (0.991 PRIMARY; 1.000 HELDOUT). Yet repeatability did not determine agreement with the program's synthetic reference labels: 1,488 of 1,920 PRIMARY judgments (77.5%) and 400 of 480 HELDOUT judgments (83.3%) matched that non-human reference. In PRIMARY, 106 cases disagreed with the synthetic reference in all four runs and three additional cases mixed match and mismatch across runs; in HELDOUT, 20 cases disagreed with the synthetic reference identically in all four runs. Separate controlled interventions on frozen 20-case bodies exposed additional measurement sensitivities. Six balanced rating-option permutations reproduced the same four-case susceptible set across two jobs, although the exact arm-level change counts differed. A material evaluator-contract revision coincided with a shift of a byte-identical 20-case body from 7 of 20 to 20 of 20 CONFIRMED on repeated evaluations; several construct instructions changed together, so no single mechanism is identified. Finally, three runs with byte-identical five-metric definitions showed 13%, 14%, and 15% pairwise judgment drift across 20 cases, whereas a separate 20-case single-metric body showed no observed categorical changes across four jobs. These findings are presented as a bounded reproducibility case study, not a new general framework. In this program, repeatability, synthetic-reference agreement, evaluator-contract sensitivity, option-order sensitivity, and observed run-to-run drift diverged enough that compressing them into one score would conceal materially different measurement behavior.