How Do Agent Benchmarks Test Underspecified Instructions? An Evidence Map of Definitions, Injection, Clarification and Scoring, 2022 to 2026

Preprint. Many agent benchmarks now test what a system does when an instruction leaves something out or admits more than one reading, and each defines the problem in its own way. This evidence map codes 215 benchmarks and datasets dated 2022 to 2026, located by searches frozen on 19 November 2026, on how they define underspecification, construct it, label it, let the agent respond, simulate the user, score the outcome, and release the data. Most are recent: 93% are dated 2025 or 2026. 66% state a definition of the problem they test. 67% create the missing information by injecting it into existing tasks or by generating tasks, and 13% use natural or mixed origins. The agent may ask a question in 71%; where it asks, a language model usually plays the user. Task success is the most common metric (75%), but 144 benchmarks also report a metric outside the common families, and only 95 state how many tasks are underspecified. Resolving the gap needs a visual or other non-text referent in 5 benchmarks according to their sources and in 14 when coder inferences are counted; it is not reported in 22. A data license is stated for 50%. Blind second coding of a seeded sample agreed with the initial coding on 90% of origin codes. The map describes the works located by these searches, not the field. AI-use disclosure: Claude (Anthropic) assisted with literature discovery, screening, data extraction, classification, analysis and manuscript preparation. The research methods and quality-control procedures are described in Section 4. The author reviewed the output.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24376785
Primary Topic
Natural Language Processing Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

How Do Agent Benchmarks Test Underspecified Instructions? An Evidence Map of Definitions, Injection, Clarification and Scoring, 2022 to 2026

Ankur Shrestha
Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
preprint

How Do Agent Benchmarks Test Underspecified Instructions? An Evidence Map of Definitions, Injection, Clarification and Scoring, 2022 to 2026

Ankur Shrestha
preprint en

Abstract

Preprint. Many agent benchmarks now test what a system does when an instruction leaves something out or admits more than one reading, and each defines the problem in its own way. This evidence map codes 215 benchmarks and datasets dated 2022 to 2026, located by searches frozen on 19 November 2026, on how they define underspecification, construct it, label it, let the agent respond, simulate the user, score the outcome, and release the data. Most are recent: 93% are dated 2025 or 2026. 66% state a definition of the problem they test. 67% create the missing information by injecting it into existing tasks or by generating tasks, and 13% use natural or mixed origins. The agent may ask a question in 71%; where it asks, a language model usually plays the user. Task success is the most common metric (75%), but 144 benchmarks also report a metric outside the common families, and only 95 state how many tasks are underspecified. Resolving the gap needs a visual or other non-text referent in 5 benchmarks according to their sources and in 14 when coder inferences are counted; it is not reported in 22. A data license is stated for 50%. Blind second coding of a seeded sample agreed with the initial coding on 90% of origin codes. The map describes the works located by these searches, not the field. AI-use disclosure: Claude (Anthropic) assisted with literature discovery, screening, data extraction, classification, analysis and manuscript preparation. The research methods and quality-control procedures are described in Section 4. The author reviewed the output.

Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon