When Reference Solutions Pass but Violate Explicit Goals: An AgentDojo Case Study
An accepted reference trajectory is often used as a check that a benchmark task can be completed. Acceptance alone does not establish agreement with an explicit user requirement. We diagnose a source-bound counterexample in AgentDojo: a travel task requests a calendar reminder on 25 April 2024, its author reference creates the event on 25 April 2023, and native utility accepts the terminal state because its date comparison discards the year. The tool faithfully persists the supplied argument. A four-cell reference-by-evaluator repair study separates the invalid reference from the year-blind grading rule. A source-guided extension covers all four calendar-writing travel references and a full-date reservation control. Two explicitly dated calendar tasks accept eighteen constructed wrong-year variants; the reservation checker rejects its nine wrong-year variants. Two requests omit the year and are not classified as year errors. Seven compatible archived actor traces in this extension show no accepted date-condition failure. Retained call/state receipts, scoped patches and fresh-copy reproduction support the diagnosis. The contribution is one empirical reference inconsistency, a two-task evaluator sensitivity result and an auditable diagnostic procedure, with no claim of prevalence or general method superiority.
Authors
- Shoufa Zhang
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-31
- DOI
- https://doi.org/10.5281/zenodo.23839672
- Primary Topic
- Software Testing and Debugging Techniques
- Type
- preprint