AI code review and static checks on 100 physics simulation repositories: what each finds, and what it takes to confirm a claim

I compared two ways of finding defects in research physics code on the same 100 public GitHub repositories: five AI code reviewers given one fixed prompt, and the static checker physverify scan. The list of known defects comes from earlier per-repository test scripts that compare the code with a physics reference. Fifty repositories had a defect I had earlier found and reported (67 defects in total); fifty had no known defect. The AI reviewers named 36 to 49 of the 67 known defects, the static checker 11. On the repositories with no known defect, I selected 119 AI claims for adjudication across five repository-based draws. For 115 of them a check script was run in a sandbox against a reference written from the physics. Of the 119, 101 received a definitive verdict: 40 were real defects that change a result, 51 were real but minor and 10 were wrong. The other 18 remained unresolved. The 40 are 33 distinct defects in 19 of the 50 "clean" repositories, so those repositories were not clean, and recall against the known list is recall against an incomplete list. The pooled counts describe the drawn claims, not the 2,673 AI claims as a whole. Nine of the ten wrong claims came from one model; of the 72 selected claims from the three Claude models, none was rejected and 12 remained undecided. All counts hold within this sample only. No human adjudication is reported; labels come from two automated checkers following written rules.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24224432
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

AI code review and static checks on 100 physics simulation repositories: what each finds, and what it takes to confirm a claim

Ram Chandra Giri
Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
preprint

AI code review and static checks on 100 physics simulation repositories: what each finds, and what it takes to confirm a claim

Ram Chandra Giri
preprint en

Abstract

I compared two ways of finding defects in research physics code on the same 100 public GitHub repositories: five AI code reviewers given one fixed prompt, and the static checker physverify scan. The list of known defects comes from earlier per-repository test scripts that compare the code with a physics reference. Fifty repositories had a defect I had earlier found and reported (67 defects in total); fifty had no known defect. The AI reviewers named 36 to 49 of the 67 known defects, the static checker 11. On the repositories with no known defect, I selected 119 AI claims for adjudication across five repository-based draws. For 115 of them a check script was run in a sandbox against a reference written from the physics. Of the 119, 101 received a definitive verdict: 40 were real defects that change a result, 51 were real but minor and 10 were wrong. The other 18 remained unresolved. The 40 are 33 distinct defects in 19 of the 50 "clean" repositories, so those repositories were not clean, and recall against the known list is recall against an incomplete list. The pooled counts describe the drawn claims, not the 2,673 AI claims as a whole. Nine of the ten wrong claims came from one model; of the 72 selected claims from the three Claude models, none was rejected and 12 remained undecided. All counts hold within this sample only. No human adjudication is reported; labels come from two automated checkers following written rules.

Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon