Valid but Wrong: Silent Enum Substitution in Tool Calls, and Why Schema Validation Cannot See It
Tool-calling systems often treat JSON-schema validation as a safety gate: if a call validates, it runs. This paper asks what that gate misses when a user gives a value that the tool's closed enum cannot express, such as "urgent-ish" for a priority field. I built 420 items across 15 tools, half awkward and half matched clean controls. Qwen2.5-7B-Instruct quietly swaps in a valid but different option on 54.8% of the awkward items, and on 34.7% of those where the closest option changes the meaning. Schema validation catches none of these. Comparing the call against the wording of the instruction catches nearly all of them, but it also flags almost every correct paraphrase. Mistral-7B and Qwen2.5-3B behave differently: they mostly repeat the invalid value, which fails loudly. But a single validation retry turns most of those failures into silent substitutions too. Letting the model ask for clarification helps the 7B model, and a hosted API that enforces the enum leaves substitution as the only option. Two trained comparators and a zero-shot judge could not reliably tell acceptable mappings from harmful ones. The study is small: three local models, one hosted model, a hand-built item set and a single annotator. The items, prompts and all model outputs are in the repository: https://github.com/LakshitF/valid-but-wrong
Authors
- Lakshit Farswan
Institutions
- The University of Texas at Austin (US)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-30
- DOI
- https://doi.org/10.5281/zenodo.23811406
- Primary Topic
- Natural Language Processing Techniques
- Type
- preprint