Test Weakening: Measuring Repair without Eroding Tests

A coding agent can make visible tests pass by changing their expected answers. Test Weakening measures whether repair preserves the evidence used to accept it. We study 30 synthetic JavaScript tasks with executable checks for original assertions, held-out behavior, and mutation strength. The study runs 11 scripted policies over 660 episodes including repeats. Reference repair earns 100.0 [88.6, 100.0] percent full success. Expected-answer rewriting earns 100.0 [88.6, 100.0] percent public-test success but 0.0 [0.0, 11.4] percent full success. Brackets are 95 percent Wilson intervals over tasks. A semantics-preserving test reformat earns 0.0 [0.0, 11.4] percent full success, exposing a lexical preservation limit. A separately labelled simulated population sweep compares public-only, original-test, and full acceptance rules. It measures scoring sensitivity under fixed behavior, not changes in learned agents. No learned model, human study, or production deployment is evaluated.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-29
DOI
https://doi.org/10.5281/zenodo.23752432
Primary Topic
Software Testing and Debugging Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Test Weakening: Measuring Repair without Eroding Tests

Pooria Arab
3 links
Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
preprint

Test Weakening: Measuring Repair without Eroding Tests

Pooria Arab
preprint en

Abstract

A coding agent can make visible tests pass by changing their expected answers. Test Weakening measures whether repair preserves the evidence used to accept it. We study 30 synthetic JavaScript tasks with executable checks for original assertions, held-out behavior, and mutation strength. The study runs 11 scripted policies over 660 episodes including repeats. Reference repair earns 100.0 [88.6, 100.0] percent full success. Expected-answer rewriting earns 100.0 [88.6, 100.0] percent public-test success but 0.0 [0.0, 11.4] percent full success. Brackets are 95 percent Wilson intervals over tasks. A semantics-preserving test reformat earns 0.0 [0.0, 11.4] percent full success, exposing a lexical preservation limit. A separately labelled simulated population sweep compares public-only, original-test, and full acceptance rules. It measures scoring sensitivity under fixed behavior, not changes in learned agents. No learned model, human study, or production deployment is evaluated.

Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon