The Perfect Score With a Missing Case: Metric Qualification Does Not Establish Evaluation-Set Conformance

A metric can be correctly computed over the records it receives while those records do not conform to the evaluation set that was declared. This paper reports a preserved evaluator-development lineage completed before target-model release. In V12, the intended nine mechanical and four human-facing measures were implemented, the all-13 self-test passed, and targeted perturbation checks passed, yet direct functional audit showed that one declared required deterministic member could be absent while deterministic_hash_equality still returned 1.0; malformed duplicate or missing grade geometries could also be accepted. The successor V13 placed declared-run identity, exact cardinality, uniqueness, required membership, presentation mapping where blinding applied, and required grade identity/cardinality ahead of metric computation. On a complete 192-run-record functional baseline, the successor gate accepted the control geometry and rejected the tested missing/duplicate attacks. The contribution is not a new checklist of validation primitives: those primitives are substantially occupied by data validation, benchmark auditing, and mature benchmark systems. The bounded result is narrower: under the qualification conditions actually observed in V12, metric qualification was insufficient to establish evaluation-set conformance. V13 is a targeted post-failure repair of the observed attack classes, not evidence of universal sufficiency, prevalence, or model-performance improvement.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-06
DOI
https://doi.org/10.5281/zenodo.22541028
Primary Topic
Adversarial Robustness in Machine Learning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
preprint

The Perfect Score With a Missing Case: Metric Qualification Does Not Establish Evaluation-Set Conformance

Logan Davis
Zenodo (CERN European Organization for Nuclear Research)
Adversarial Robustness in Machine Learning
preprint

The Perfect Score With a Missing Case: Metric Qualification Does Not Establish Evaluation-Set Conformance

Logan Davis
preprint en

Abstract

A metric can be correctly computed over the records it receives while those records do not conform to the evaluation set that was declared. This paper reports a preserved evaluator-development lineage completed before target-model release. In V12, the intended nine mechanical and four human-facing measures were implemented, the all-13 self-test passed, and targeted perturbation checks passed, yet direct functional audit showed that one declared required deterministic member could be absent while deterministic_hash_equality still returned 1.0; malformed duplicate or missing grade geometries could also be accepted. The successor V13 placed declared-run identity, exact cardinality, uniqueness, required membership, presentation mapping where blinding applied, and required grade identity/cardinality ahead of metric computation. On a complete 192-run-record functional baseline, the successor gate accepted the control geometry and rejected the tested missing/duplicate attacks. The contribution is not a new checklist of validation primitives: those primitives are substantially occupied by data validation, benchmark auditing, and mature benchmark systems. The bounded result is narrower: under the qualification conditions actually observed in V12, metric qualification was insufficient to establish evaluation-set conformance. V13 is a targeted post-failure repair of the observed attack classes, not evidence of universal sufficiency, prevalence, or model-performance improvement.

Zenodo (CERN European Organization for Nuclear Research)
Logan Regional Hospital (US)
Adversarial Robustness in Machine Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

The Perfect Score With a Missing Case: Metric Qualification Does Not Establish Evaluation-Set Conformance — Logan Davis · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS