When Tests Need an Answer: Oracle Underspecification Suppresses Diagnostic Exploration in LLM-Generated Verification

Executable tests generated from natural-language requirements require definite expected outputs. When a requirement leaves an oracle unresolved, a generator can guess, abstain, or avoid inputs on which the unresolved choice matters. This controlled study tests whether requiring a definite executable oracle suppresses diagnostic-input selection under requirement underspecification, separates the effect from edge-case salience, and evaluates an explicit UNRESOLVED output channel. Across 12 finite-domain tasks in four semantic families, salience-matched ambiguity produced 0/192 diagnostic selections versus 52/96 after clarification. Allowing UNRESOLVED restored 69/192 diagnostic selections, although mitigation precision was 69/79 (87.3%), below the preregistered 90% gate. The study identifies a bounded oracle-coupled selection failure mode rather than estimating its prevalence.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-05
DOI
https://doi.org/10.5281/zenodo.22379282
Primary Topic
Natural Language Processing Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

When Tests Need an Answer: Oracle Underspecification Suppresses Diagnostic Exploration in LLM-Generated Verification

Haitam Kadri
Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
preprint

When Tests Need an Answer: Oracle Underspecification Suppresses Diagnostic Exploration in LLM-Generated Verification

Haitam Kadri
preprint en

Abstract

Executable tests generated from natural-language requirements require definite expected outputs. When a requirement leaves an oracle unresolved, a generator can guess, abstain, or avoid inputs on which the unresolved choice matters. This controlled study tests whether requiring a definite executable oracle suppresses diagnostic-input selection under requirement underspecification, separates the effect from edge-case salience, and evaluates an explicit UNRESOLVED output channel. Across 12 finite-domain tasks in four semantic families, salience-matched ambiguity produced 0/192 diagnostic selections versus 52/96 after clarification. Allowing UNRESOLVED restored 69/192 diagnostic selections, although mitigation precision was 69/79 (87.3%), below the preregistered 90% gate. The study identifies a bounded oracle-coupled selection failure mode rather than estimating its prevalence.

Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.