Chaos Monkey for Coding Agents: Measuring How Coding-Agent Harnesses Recover When Their Tools Fail

Coding agents run inside harnesses that execute their tool calls, and those tools fail: commands error, reads return nothing or stale data, and calls that succeeded report failure. We present Chaos Monkey for Coding Agents (CMCA), an open benchmark that applies one seeded fault model at two layers: an MCP proxy in front of mock services, and injectors loaded through each harness's own hooks that fault its built-in read, edit, search and shell tools, restricted to fault types a conformance suite shows each harness delivers. Every configuration uses the same main model, so rows compare harness configurations; the native-track fx row is fxh, a fork that adds tool hooks. Besides success under faults, CMCA reports whether agents state their outcome honestly, whether stale data reaches their output, and what recovery costs. On 37 service and 36 native tasks (3,011 trial runs), fx leads the service track (83 of 111 faulted trials) and fxh the native track (95 of 108), but no pairwise difference survives Holm correction. Among failed trials that received a fault, agents still reported success in 60% to 94% of cases. Code, tasks, trajectories and analysis are public. Results are release 2026-11-18-m6 (git tag m6-2026-11-18). Results site: https://www.chaos-monkey.tech

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-20
DOI
https://doi.org/10.5281/zenodo.24403111
Primary Topic
Software Testing and Debugging Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Chaos Monkey for Coding Agents: Measuring How Coding-Agent Harnesses Recover When Their Tools Fail

Daniel Ávila Arias
2 links
Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
preprint

Chaos Monkey for Coding Agents: Measuring How Coding-Agent Harnesses Recover When Their Tools Fail

Daniel Ávila Arias
preprint en

Abstract

Coding agents run inside harnesses that execute their tool calls, and those tools fail: commands error, reads return nothing or stale data, and calls that succeeded report failure. We present Chaos Monkey for Coding Agents (CMCA), an open benchmark that applies one seeded fault model at two layers: an MCP proxy in front of mock services, and injectors loaded through each harness's own hooks that fault its built-in read, edit, search and shell tools, restricted to fault types a conformance suite shows each harness delivers. Every configuration uses the same main model, so rows compare harness configurations; the native-track fx row is fxh, a fork that adds tool hooks. Besides success under faults, CMCA reports whether agents state their outcome honestly, whether stale data reaches their output, and what recovery costs. On 37 service and 36 native tasks (3,011 trial runs), fx leads the service track (83 of 111 faulted trials) and fxh the native track (95 of 108), but no pairwise difference survives Holm correction. Among failed trials that received a fault, agents still reported success in 60% to 94% of cases. Code, tasks, trajectories and analysis are public. Results are release 2026-11-18-m6 (git tag m6-2026-11-18). Results site: https://www.chaos-monkey.tech

Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon