Why Sense Matters: Answer-Key Leakage and the Specification Boundary in Agentic Citation Verification

Agentic evaluation workflows increasingly rely on loops: models call tools, retrieve references, receive feedback, pass through gates, and are checked by other agents or scripts. Such loops can verify many specified properties, but they do not define their own validity boundary. This paper argues that agentic evaluation is specification-bound before it is metric-bound: metrics are interpretable only after the evaluated object has been correctly specified. The argument is developed through a diagnostic case in LLM-based citation verification. A multi-round audit process recomputed metrics, checked trial counts, inspected scripts, and found several local inconsistencies, yet initially missed the decisive validity threat: model-visible reference records contained scoring-only answer labels, including has_error and error_type. After model-visible references were separated from scoring-only ground truth, apparent GATE advantages collapsed: DeepSeek V3 GATE dropped from Youden's J = +0.659 to +0.015, and Grok GATE dropped from +0.842 to +0.105. A clean six-cell rerun is reported only as secondary diagnostic evidence, not as a model-ranking benchmark. The paper introduces the specification-boundary framework and distinguishes metric audit from condition-definition audit. Metric audit asks whether reported numbers follow from trial files. Condition-definition audit asks whether prompts, tools, files, memories, retrieval contexts, and reference stores instantiate the intended experimental condition. The prior capacity to identify validity threats, define information boundaries, and translate them into auditable constraints is called anticipatory specification judgment (ASJ). In agentic evaluation, asking whether a result "makes sense" is asking whether the underlying specification admits a coherent evaluable object.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-06-16
DOI
https://doi.org/10.5281/zenodo.20710741
Citations
2
Primary Topic
Scientific Computing and Data Management
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Why Sense Matters: Answer-Key Leakage and the Specification Boundary in Agentic Citation Verification

Jianeng Zhou
2 citations
Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
preprint

Why Sense Matters: Answer-Key Leakage and the Specification Boundary in Agentic Citation Verification

Jianeng Zhou
preprint en
2 citations

Abstract

Agentic evaluation workflows increasingly rely on loops: models call tools, retrieve references, receive feedback, pass through gates, and are checked by other agents or scripts. Such loops can verify many specified properties, but they do not define their own validity boundary. This paper argues that agentic evaluation is specification-bound before it is metric-bound: metrics are interpretable only after the evaluated object has been correctly specified. The argument is developed through a diagnostic case in LLM-based citation verification. A multi-round audit process recomputed metrics, checked trial counts, inspected scripts, and found several local inconsistencies, yet initially missed the decisive validity threat: model-visible reference records contained scoring-only answer labels, including has_error and error_type. After model-visible references were separated from scoring-only ground truth, apparent GATE advantages collapsed: DeepSeek V3 GATE dropped from Youden's J = +0.659 to +0.015, and Grok GATE dropped from +0.842 to +0.105. A clean six-cell rerun is reported only as secondary diagnostic evidence, not as a model-ranking benchmark. The paper introduces the specification-boundary framework and distinguishes metric audit from condition-definition audit. Metric audit asks whether reported numbers follow from trial files. Condition-definition audit asks whether prompts, tools, files, memories, retrieval contexts, and reference stores instantiate the intended experimental condition. The prior capacity to identify validity threats, define information boundaries, and translate them into auditable constraints is called anticipatory specification judgment (ASJ). In agentic evaluation, asking whether a result "makes sense" is asking whether the underlying specification admits a coherent evaluable object.

Zenodo (CERN European Organization for Nuclear Research)
Shantou University (CN)
Peace, Justice and strong institutions
Scientific Computing and Data Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.