When Reference Solutions Pass but Violate Explicit Goals: An AgentDojo Case Study

An accepted reference trajectory is often used as a check that a benchmark task can be completed. Acceptance alone does not establish agreement with an explicit user requirement. We diagnose a source-bound counterexample in AgentDojo: a travel task requests a calendar reminder on 25 April 2024, its author reference creates the event on 25 April 2023, and native utility accepts the terminal state because its date comparison discards the year. The tool faithfully persists the supplied argument. A four-cell reference-by-evaluator repair study separates the invalid reference from the year-blind grading rule. A source-guided extension covers all four calendar-writing travel references and a full-date reservation control. Two explicitly dated calendar tasks accept eighteen constructed wrong-year variants; the reservation checker rejects its nine wrong-year variants. Two requests omit the year and are not classified as year errors. Seven compatible archived actor traces in this extension show no accepted date-condition failure. Retained call/state receipts, scoped patches and fresh-copy reproduction support the diagnosis. The contribution is one empirical reference inconsistency, a two-task evaluator sensitivity result and an auditable diagnostic procedure, with no claim of prevalence or general method superiority.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-31
DOI
https://doi.org/10.5281/zenodo.23839672
Primary Topic
Software Testing and Debugging Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

When Reference Solutions Pass but Violate Explicit Goals: An AgentDojo Case Study

Shoufa Zhang
Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
preprint

When Reference Solutions Pass but Violate Explicit Goals: An AgentDojo Case Study

Shoufa Zhang
preprint en

Abstract

An accepted reference trajectory is often used as a check that a benchmark task can be completed. Acceptance alone does not establish agreement with an explicit user requirement. We diagnose a source-bound counterexample in AgentDojo: a travel task requests a calendar reminder on 25 April 2024, its author reference creates the event on 25 April 2023, and native utility accepts the terminal state because its date comparison discards the year. The tool faithfully persists the supplied argument. A four-cell reference-by-evaluator repair study separates the invalid reference from the year-blind grading rule. A source-guided extension covers all four calendar-writing travel references and a full-date reservation control. Two explicitly dated calendar tasks accept eighteen constructed wrong-year variants; the reservation checker rejects its nine wrong-year variants. Two requests omit the year and are not classified as year errors. Seven compatible archived actor traces in this extension show no accepted date-condition failure. Retained call/state receipts, scoped patches and fresh-copy reproduction support the diagnosis. The contribution is one empirical reference inconsistency, a two-task evaluator sensitivity result and an auditable diagnostic procedure, with no claim of prevalence or general method superiority.

Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
When Reference Solutions Pass but Violate Explicit Goals: An AgentDojo Case Study — Shoufa Zhang · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS