When Tests Need an Answer: Oracle Underspecification Suppresses Diagnostic Exploration in LLM-Generated Verification
Executable tests generated from natural-language requirements require definite expected outputs. When a requirement leaves an oracle unresolved, a generator can guess, abstain, or avoid inputs on which the unresolved choice matters. This controlled study tests whether requiring a definite executable oracle suppresses diagnostic-input selection under requirement underspecification, separates the effect from edge-case salience, and evaluates an explicit UNRESOLVED output channel. Across 12 finite-domain tasks in four semantic families, salience-matched ambiguity produced 0/192 diagnostic selections versus 52/96 after clarification. Allowing UNRESOLVED restored 69/192 diagnostic selections, although mitigation precision was 69/79 (87.3%), below the preregistered 90% gate. The study identifies a bounded oracle-coupled selection failure mode rather than estimating its prevalence.
Authors
- Haitam Kadri (ORCID: https://orcid.org/0009-0001-3791-1401)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-09-05
- DOI
- https://doi.org/10.5281/zenodo.22379283
- Primary Topic
- Natural Language Processing Techniques
- Type
- preprint