Behavioural Drift in Model-Checker-Accepted Repairs: A Measurement and Witness Audit

A model checker can accept a memory-safety repair even when the repaired program produces a different result. We re-evaluated previously accepted FormAI repairs on held-out inputs that passed the original-program safety screens and met a mechanical test of input well-formedness. Observable behaviour changed for 24/86 repairs (27.9%; program-cluster 95% interval 15.1–42.4%), across 12 of 29 eligible programs. Replaying the tests without an output limit preserved this count. An AI-assisted audit classified 2/86 repairs as likely regressions in computed output; independent human adjudication was unavailable. Other changes included input rejection and corrections to an erroneous original. On contest programs, the combined corpus exposed changes in 63/112 repairs, compared with 0/112 on retained official tests and 0/19 on dataset-generated tests. Some inputs that met the mechanical rule fell outside the problem domain. These results show how strongly the estimated rate of behavioural change depends on the test inputs and the evidence available for judging the original output.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24374772
Primary Topic
Software Testing and Debugging Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Behavioural Drift in Model-Checker-Accepted Repairs: A Measurement and Witness Audit

S M Abdulla Al Mamun
Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
preprint

Behavioural Drift in Model-Checker-Accepted Repairs: A Measurement and Witness Audit

S M Abdulla Al Mamun
preprint en

Abstract

A model checker can accept a memory-safety repair even when the repaired program produces a different result. We re-evaluated previously accepted FormAI repairs on held-out inputs that passed the original-program safety screens and met a mechanical test of input well-formedness. Observable behaviour changed for 24/86 repairs (27.9%; program-cluster 95% interval 15.1–42.4%), across 12 of 29 eligible programs. Replaying the tests without an output limit preserved this count. An AI-assisted audit classified 2/86 repairs as likely regressions in computed output; independent human adjudication was unavailable. Other changes included input rejection and corrections to an erroneous original. On contest programs, the combined corpus exposed changes in 63/112 repairs, compared with 0/112 on retained official tests and 0/19 on dataset-generated tests. Some inputs that met the mechanical rule fell outside the problem domain. These results show how strongly the estimated rate of behavioural change depends on the test inputs and the evidence available for judging the original output.

Zenodo (CERN European Organization for Nuclear Research)
University of Manchester (GB)
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon