Make the Reviewer Run the Test: Team Pressure, Bug-Claim Deference and a Test-First Fix in Gemma 4 Code-Review Agents
Preregistered Study 5 and exploratory Study 5b in the BEP Principle Series. Gemma 4 E4B-it, Gemma 4 12B-it and Grok 4.20 reviewed 64 Python functions (verified correct and buggy versions) in 128 minimal-pair scenarios as members of a multi-agent software team whose membership was secure, conditional on agreement, or absent (1,152 reviews). The preregistered hypothesis that conditional membership increases deference was not supported (0/3 models); team membership increased deference relative to no team in both Gemma models (2/3), reported as exploratory because judge-human agreement (kappa = 0.76) fell below the preregistered 0.80. Agents almost never approved buggy code when CI failed (1 of 288), but rejected correct code 88-100% of the time when the team claimed a bug and CI passed. In Study 5b, a test-first Gemma agent that wrote tests, had them executed against the code and then decided cut those false rejections from 90% to 10% (E4B) and from 98% to 31% (12B). Costs: E4B approved more buggy code when its own tests missed the bug, and Gemma 12B under conditional membership still rejected half of the correct functions, often after its own tests passed. A second judge (Gemma 4 12B) agreed with the first at kappa = 0.98. All data, code and run logs are included. Submitted to the Kaggle Gemma 4 Developer Agent Paper Track.
Authors
- Nicolas Brian Quiroz
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-01
- DOI
- https://doi.org/10.5281/zenodo.23874136
- Primary Topic
- Software Engineering Research
- Type
- preprint