Scorer Defects After the Result

A controlled 288-run evaluation of one coded decision-state slice produced a reproducible as-scored disposition under its original scoring contract. The study contained 144 runs per arm and 864 completed turns, with blind grading and a project-recorded logistic mixed-effects analysis whose frozen code is now directly reproducible. Observed governed-task success was 19/144 in Arm A and 23/144 in Arm B. Marginal mixed-model estimates were 0.1316 and 0.1591, respectively, for a +2.75 percentage-point contrast (95% CI -4.12 to +9.62 pp). A recovered project custody record states a +10 percentage-point practical-effect criterion with a positive lower confidence bound; no independently inspectable pre-result artifact establishing that criterion was recovered. Under that project-recorded criterion, the as-scored coded slice did not satisfy the stated practical-effect requirement. Measured Answer-Key Burden was also higher in Arm B. After the primary score and analysis were frozen, a read-only audit found a measurement-contract defect in the deterministic `unsupported_claims` scorer: any nonempty list failed, although the field lacked a precise normative definition and many entries were descriptive withholding or refusal language rather than clearly asserted unsupported claims. The audit did not establish authoritative replacement labels. Historical scores were therefore preserved as the record of the scoring contract actually used. Two deliberately extreme post-result reclassification scenarios over sole blockers moved the descriptive B-A contrast to -5.56 percentage points in both cases; they are illustrations, not corrected treatment effects or formal bounds. This manuscript reports the historical result, the later instrument diagnosis, and the post-result reclassification scenarios as separate evidence states. It does not claim a new general evaluation-repair method, independently proven preregistration, or a corrected efficacy estimate.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-05
DOI
https://doi.org/10.5281/zenodo.22343212
Primary Topic
Meta-analysis and systematic reviews
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Scorer Defects After the Result

Logan Davis
Zenodo (CERN European Organization for Nuclear Research)
Meta-analysis and systematic reviews
preprint

Scorer Defects After the Result

Logan Davis
preprint en

Abstract

A controlled 288-run evaluation of one coded decision-state slice produced a reproducible as-scored disposition under its original scoring contract. The study contained 144 runs per arm and 864 completed turns, with blind grading and a project-recorded logistic mixed-effects analysis whose frozen code is now directly reproducible. Observed governed-task success was 19/144 in Arm A and 23/144 in Arm B. Marginal mixed-model estimates were 0.1316 and 0.1591, respectively, for a +2.75 percentage-point contrast (95% CI -4.12 to +9.62 pp). A recovered project custody record states a +10 percentage-point practical-effect criterion with a positive lower confidence bound; no independently inspectable pre-result artifact establishing that criterion was recovered. Under that project-recorded criterion, the as-scored coded slice did not satisfy the stated practical-effect requirement. Measured Answer-Key Burden was also higher in Arm B. After the primary score and analysis were frozen, a read-only audit found a measurement-contract defect in the deterministic `unsupported_claims` scorer: any nonempty list failed, although the field lacked a precise normative definition and many entries were descriptive withholding or refusal language rather than clearly asserted unsupported claims. The audit did not establish authoritative replacement labels. Historical scores were therefore preserved as the record of the scoring contract actually used. Two deliberately extreme post-result reclassification scenarios over sole blockers moved the descriptive B-A contrast to -5.56 percentage points in both cases; they are illustrations, not corrected treatment effects or formal bounds. This manuscript reports the historical result, the later instrument diagnosis, and the post-result reclassification scenarios as separate evidence states. It does not claim a new general evaluation-repair method, independently proven preregistration, or a corrected efficacy estimate.

Zenodo (CERN European Organization for Nuclear Research)
Logan Regional Hospital (US)
Gender equality, Peace, Justice and strong institutions
Meta-analysis and systematic reviews
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.