When Does Asynchronous Review Catch an AI Agent in Time? A Simulation Study of Monitoring under Staffing Gaps, Auditor Error, Bursts and Load
Reproduction package and preprint for "When Does Asynchronous Review Catch an AI Agent in Time?" (ZannixLabs report ZLX-AIS-2026-W11, revision 4). Not peer reviewed. Asynchronous monitoring lets AI agents act immediately while a monitor scores their actions and human auditors review the most suspicious ones afterwards. It prevents harm only if review, and the response to it, finishes before an attack takes effect. We re-implement the deployment model from the UK AI Security Institute's Async Control study as a discrete-event simulator, replicate the authors' reference code (largest deviation |z| = 1.68 across four settings), and extend it to staffing schedules, auditor misses, monitor evasion, correlated bursts, a response delay and an uncertain upper tail of honest scores. Key results: with auditors on duty around the clock, 95.8% of successful attacks are caught before a 60-minute harm latency; the same weekly audit budget concentrated into business hours catches 3.3% against an attacker who acts when no one is on duty. For bursts of simultaneous attacks, the halt rule decides the outcome: with a 30-minute fleet-wide halt triggered by any caught attack, bursts of 8 and 16 were all stopped; if only a caught successful attack triggers it, about 90% are. Whether traffic load matters depends on the extreme tail of honest monitor scores, which 451 honest episodes cannot pin down. The reference simulator also front-loads fractional audit capacity, which inflates safety at low audit rates (86.7% versus 33.1% with capacity spread evenly). Contents:- async-monitoring-oversight-latency.pdf: the paper (CC BY 4.0).- async-monitoring-oversight-latency.tar.gz: the reproduction package. It holds the unmodified upstream source file (00_raw), extraction script, simulator, every result table with all configurations and seeds, the pre-registration and deviations log, automated tests of the model properties, validation scripts against the reference implementation, and the paper source. To reproduce: unpack the archive and run "python reproduce.py" (about 25 minutes on 2 cores) or "python reproduce.py --quick" for a smoke test. SHA256SUMS.txt lists checksums for every file. The simulator, analysis code and drafts were produced with assistance from Claude (Anthropic); the author directed the study and is responsible for its content. The paper has been through several rounds of AI-assisted review, not peer review.
Authors
- Ajmal Baba
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-14
- DOI
- https://doi.org/10.5281/zenodo.24226856
- Primary Topic
- Adversarial Robustness in Machine Learning
- Type
- article
- Field-Weighted Citation Impact
- 0.00