AI code review and static checks on 100 physics simulation repositories: what each finds, and what it takes to confirm a claim
I compared two ways of finding defects in research physics code on the same 100 public GitHub repositories: five AI code reviewers given one fixed prompt, and the static checker physverify scan. The list of known defects comes from earlier per-repository test scripts that compare the code with a physics reference. Fifty repositories had a defect I had earlier found and reported (67 defects in total); fifty had no known defect. The AI reviewers named 36 to 49 of the 67 known defects, the static checker 11. On the repositories with no known defect, I selected 119 AI claims for adjudication across five repository-based draws. For 115 of them a check script was run in a sandbox against a reference written from the physics. Of the 119, 101 received a definitive verdict: 40 were real defects that change a result, 51 were real but minor and 10 were wrong. The other 18 remained unresolved. The 40 are 33 distinct defects in 19 of the 50 "clean" repositories, so those repositories were not clean, and recall against the known list is recall against an incomplete list. The pooled counts describe the drawn claims, not the 2,673 AI claims as a whole. Nine of the ten wrong claims came from one model; of the 72 selected claims from the three Claude models, none was rejected and 12 remained undecided. All counts hold within this sample only. No human adjudication is reported; labels come from two automated checkers following written rules.
Authors
- Ram Chandra Giri (ORCID: https://orcid.org/0009-0002-9763-9910)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-14
- DOI
- https://doi.org/10.5281/zenodo.24224432
- Primary Topic
- Software Engineering Research
- Type
- preprint