What the Log Misses: Coding Agents Break Passing Tests Mid-Run, Often Unseen

On the way to fixing an issue, a coding agent often breaks code that used to work, sometimes without noticing. Two records could reveal such a break, but neither does reliably. The final patch, which leaderboards grade, keeps no trace of a break repaired before submission. The log, which model judges and reward models read, shows a break only if the agent runs something that exposes it. To measure what they miss, we rerun the published commands of 41 entries on SWE-bench Verified's bash-only leaderboard for 40 tasks, test the code after each command, and check whether the log showed each break. Over a third of runs break a test that passed at the start, and about half of these breaks never show in the log. Most go unseen because agents check their fix with their own scripts, as their instructions suggest, and rarely run the repository's tests. When they do, the log shows a failing test nine times in ten. One run in twelve finishes with a break its log never showed. Breaks are as common on SWE-bench Multilingual, where most go unseen. Judges that read the log catch most breaks, but the tests confirm only about two in five of the best judges' alarms, so a judge can decide when to run tests but cannot replace them. Rerunning published commands, by contrast, needs no model calls and costs about $15, so leaderboards can report these breaks, and judges and reward models can be checked against them.