Does the Official Code-Review Plugin Still Help? A First Look, Comparing It with a Simple Request on Two New AI Models

AI coding tools can be given add-ons called plugins. A plugin is a set of written steps that tells the AI how to do a job. The steps are written for the AI models of the day. Then a newer model comes out, and few people check whether the steps still help. This paper checks one well-known plugin: the official code-review plugin for Claude Code, made by Anthropic. We gave the AI 15 real code changes to check for mistakes. Each change was checked two ways: with the plugin, and with a simple one-paragraph request. We did this on two models, Claude Opus 5.5 and Claude Fable 5.2, twice each. That made 120 reviews. We kept a record of which model answered every request. Without the plugin, each model found about twice as many of the known mistakes. It found 66% against 31% on Opus 5.5, and 73% against 32% on Fable 5.2. The plugin said much less. A bigger share of what it said matched a known mistake, and it cost about one and a half times as much. On none of the 15 code changes did the plugin find more of the known mistakes than the simple request. The plugin’s one advantage was smaller on the newer model. The plugin also picks its own models for its helpers. So on the newer model, the checking was still done by the older one. And the older model skipped the plugin’s team process in 13 of 30 reviews. Those reviews cost a third as much and did about as well. We also looked at the 543 things the AI reported that were not on the list of known mistakes. A second AI judged 285 of them to be real. They come to 123 separate mistakes the list does not have. When a more careful AI review looked again at 30 of those judgements, it agreed with 24. This is a small study. The test set is public, an AI did the grading, and we did not set pass marks ahead of time. The results are a first look. They call for a bigger study. They are not a final verdict on the plugin.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24357017
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Does the Official Code-Review Plugin Still Help? A First Look, Comparing It with a Simple Request on Two New AI Models

Ian J Michel
Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
preprint

Does the Official Code-Review Plugin Still Help? A First Look, Comparing It with a Simple Request on Two New AI Models

Ian J Michel
preprint en

Abstract

AI coding tools can be given add-ons called plugins. A plugin is a set of written steps that tells the AI how to do a job. The steps are written for the AI models of the day. Then a newer model comes out, and few people check whether the steps still help. This paper checks one well-known plugin: the official code-review plugin for Claude Code, made by Anthropic. We gave the AI 15 real code changes to check for mistakes. Each change was checked two ways: with the plugin, and with a simple one-paragraph request. We did this on two models, Claude Opus 5.5 and Claude Fable 5.2, twice each. That made 120 reviews. We kept a record of which model answered every request. Without the plugin, each model found about twice as many of the known mistakes. It found 66% against 31% on Opus 5.5, and 73% against 32% on Fable 5.2. The plugin said much less. A bigger share of what it said matched a known mistake, and it cost about one and a half times as much. On none of the 15 code changes did the plugin find more of the known mistakes than the simple request. The plugin’s one advantage was smaller on the newer model. The plugin also picks its own models for its helpers. So on the newer model, the checking was still done by the older one. And the older model skipped the plugin’s team process in 13 of 30 reviews. Those reviews cost a third as much and did about as well. We also looked at the 543 things the AI reported that were not on the list of known mistakes. A second AI judged 285 of them to be real. They come to 123 separate mistakes the list does not have. When a more careful AI review looked again at 30 of those judgements, it agreed with 24. This is a small study. The test set is public, an AI did the grading, and we did not set pass marks ahead of time. The results are a first look. They call for a bigger study. They are not a final verdict on the plugin.

Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon