Valid but Wrong: Silent Enum Substitution in Tool Calls, and Why Schema Validation Cannot See It

Tool-calling systems often treat JSON-schema validation as a safety gate: if a call validates, it runs. This paper asks what that gate misses when a user gives a value that the tool's closed enum cannot express, such as "urgent-ish" for a priority field. I built 420 items across 15 tools, half awkward and half matched clean controls. Qwen2.5-7B-Instruct quietly swaps in a valid but different option on 54.8% of the awkward items, and on 34.7% of those where the closest option changes the meaning. Schema validation catches none of these. Comparing the call against the wording of the instruction catches nearly all of them, but it also flags almost every correct paraphrase. Mistral-7B and Qwen2.5-3B behave differently: they mostly repeat the invalid value, which fails loudly. But a single validation retry turns most of those failures into silent substitutions too. Letting the model ask for clarification helps the 7B model, and a hosted API that enforces the enum leaves substitution as the only option. Two trained comparators and a zero-shot judge could not reliably tell acceptable mappings from harmful ones. The study is small: three local models, one hosted model, a hand-built item set and a single annotator. The items, prompts and all model outputs are in the repository: https://github.com/LakshitF/valid-but-wrong

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-30
DOI
https://doi.org/10.5281/zenodo.23811406
Primary Topic
Natural Language Processing Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Valid but Wrong: Silent Enum Substitution in Tool Calls, and Why Schema Validation Cannot See It

Lakshit Farswan
2 links
Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
preprint

Valid but Wrong: Silent Enum Substitution in Tool Calls, and Why Schema Validation Cannot See It

Lakshit Farswan
preprint en

Abstract

Tool-calling systems often treat JSON-schema validation as a safety gate: if a call validates, it runs. This paper asks what that gate misses when a user gives a value that the tool's closed enum cannot express, such as "urgent-ish" for a priority field. I built 420 items across 15 tools, half awkward and half matched clean controls. Qwen2.5-7B-Instruct quietly swaps in a valid but different option on 54.8% of the awkward items, and on 34.7% of those where the closest option changes the meaning. Schema validation catches none of these. Comparing the call against the wording of the instruction catches nearly all of them, but it also flags almost every correct paraphrase. Mistral-7B and Qwen2.5-3B behave differently: they mostly repeat the invalid value, which fails loudly. But a single validation retry turns most of those failures into silent substitutions too. Letting the model ask for clarification helps the 7B model, and a hosted API that enforces the enum leaves substitution as the only option. Two trained comparators and a zero-shot judge could not reliably tell acceptable mappings from harmful ones. The study is small: three local models, one hosted model, a hand-built item set and a single annotator. The items, prompts and all model outputs are in the repository: https://github.com/LakshitF/valid-but-wrong

Zenodo (CERN European Organization for Nuclear Research)
The University of Texas at Austin (US)
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Valid but Wrong: Silent Enum Substitution in Tool Calls, and Why Schema Validation Cannot See It — Lakshit Farswan · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS