CI-DeceptiveTest: An Execution-Paired Benchmark for Test Assurance in AI-Assisted Development
Abstract AI-assisted development weakens the meaning of a green continuous-integration (CI) check: generated tests may execute realistic code and pass while replacing semantic oracles with tautologies or coarse smoke checks. We introduce CI-DeceptiveTest, a Python execution-paired benchmark and Gatekeeper protocol for detecting such low-assurance tests. Its 600 pairs originate from real tests in 12 open-source repositories; each source-control test is paired with a candidate produced by controlled oracle weakening, yielding 1200 label-free static-binary views. Construction combines deterministic validation, execution evidence, a 20% manually annotated slice, random inspection of 30% of AI-assisted records. The evidence is GPT-5.5-first: GPT-5.5 supports construction, review, and full-dataset validation. Labels are transformation-defined, not model-assigned. On the full benchmark, transparent static rules achieve 0.928 low-assurance recall but a 0.458 false-block rate on accepted source-control tests. mimo-v2.5-pro is retained as an auxiliary external cross-check; it reaches 0.885 macro F1 and reduces false blocking to 0.048. Bounded mutation analysis over 42 pre-specified source/candidate pairs further supports the paired construction. The central stress result is that weak-oracle detection remains hard: plausible coarse assertions retain execution while losing semantic constraint. The benchmark enables CI gatekeeper evaluation in a controlled, reproducible setting.
Authors
- Qining Li
Institutions
- Zhejiang University of Science and Technology (CN)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-13
- DOI
- https://doi.org/10.21203/rs.3.rs-11393646/v1
- Primary Topic
- Software Testing and Debugging Techniques
- Type
- preprint