When the Total Is Right: Can small language models write tests for errors in payment reports?

Testing a financial report requires two judgments. A reviewer must choose transactions on which a faulty report will differ from a correct one, then calculate the output that the correct report should produce. We evaluated these judgments using two fictional payment policies, two local language models, and 40 reviewer responses. In the settlement-day task, none of 20 responses satisfied the strict test contract. An exploratory replay recovered five Qwen inputs that exposed the fault, but all five predicted an incorrect breakdown. In the merchant-attribution task, seven of 20 responses supplied scorable inputs. All seven exposed the fault; five also predicted the correct result. These pilots illustrate why evaluations of generated tests should report input quality and expected-output accuracy separately. They provide limited evidence about the effect of showing reviewers the implementation.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24352312
Primary Topic
Software Testing and Debugging Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

When the Total Is Right: Can small language models write tests for errors in payment reports?

Sai Charith Reddy Yaredla
3 links
Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
preprint

When the Total Is Right: Can small language models write tests for errors in payment reports?

Sai Charith Reddy Yaredla
preprint en

Abstract

Testing a financial report requires two judgments. A reviewer must choose transactions on which a faulty report will differ from a correct one, then calculate the output that the correct report should produce. We evaluated these judgments using two fictional payment policies, two local language models, and 40 reviewer responses. In the settlement-day task, none of 20 responses satisfied the strict test contract. An exploratory replay recovered five Qwen inputs that exposed the fault, but all five predicted an incorrect breakdown. In the merchant-attribution task, seven of 20 responses supplied scorable inputs. All seven exposed the fault; five also predicted the correct result. These pilots illustrate why evaluations of generated tests should report input quality and expected-output accuracy separately. They provide limited evidence about the effect of showing reviewers the implementation.

Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon