When the Total Is Right: Can small language models write tests for errors in payment reports?
Testing a financial report requires two judgments. A reviewer must choose transactions on which a faulty report will differ from a correct one, then calculate the output that the correct report should produce. We evaluated these judgments using two fictional payment policies, two local language models, and 40 reviewer responses. In the settlement-day task, none of 20 responses satisfied the strict test contract. An exploratory replay recovered five Qwen inputs that exposed the fault, but all five predicted an incorrect breakdown. In the merchant-attribution task, seven of 20 responses supplied scorable inputs. All seven exposed the fault; five also predicted the correct result. These pilots illustrate why evaluations of generated tests should report input quality and expected-output accuracy separately. They provide limited evidence about the effect of showing reviewers the implementation.
Authors
- Sai Charith Reddy Yaredla
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24352312
- Primary Topic
- Software Testing and Debugging Techniques
- Type
- preprint