Do AI Agents Need Mentors? Evaluating Chain-Pattern Interrupt (CPI) for Oversight and Reliability
<p>AI agents tackling complex, long-horizon tasks can get trapped in reasoning lock-in (RLI), where small misconceptions early in the task cascade into errors, wasted tokens and destructive outcomes. We introduce Chain-Pattern Interrupt (CPI), an external mentor mechanism that 'pauses' the agent at uncertainty or hazard points and elicits a mentor-like re-evaluation before continuing. We evaluate on two adversarial benchmarks: a debugging scenario and a priority-conflict scenario. With CPI, agents consistently deliver the requested outputs, avoid misleading suggestions, and roughly double success while reducing harmful actions by half. Our evaluation harness, logs, audit trails and replication instructions are released, enabling full reproducibility. Across 153 runs, success increased from 27% to 54% and harmful actions fell substantially; pooled OR 1.98 (CMH = 0.099). We show confidence intervals and full tests in Appendix B.</p>
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2025-01-01
- DOI
- https://doi.org/10.13140/rg.2.2.18237.93922
- Primary Topic
- Geochemistry and Geologic Mapping
- Type
- article
- Field-Weighted Citation Impact
- 0.00