When May an Agent Act Alone? An Anytime-Valid Gate on a Measured Competence Profile
Language-model agents decide, at every step of a long task, whether to act alone or to have the step checked first. We base this decision on a measured record, not on the model's self-assessment: a competence profile holding, per topic, how often an external verifier found the engine right, wrong, or could not check it. A gate lets steps on a topic proceed unchecked only after a likelihood-ratio test martingale rejects "the contradiction rate on this topic is at least the ceiling". The test is classical; the protocol around it is the contribution: pooling over complete rounds of an abstract type opens sparse topics earlier, a seeded random audit tests the unchecked steps and closes a topic whose rate rises, and a block rule handles dependence inside a task. Proposition 1 bounds, with probability at least 1-delta-delta_eta, for every topic at once, at every time of a run whose rates do not drift and under the planner's own stopping rule, the probability that a checkable result executed unchecked is wrong: at most epsilon plus one stated term per broken assumption (predictor miss, distribution shift, within-type dispersion). Under drift the bound becomes a detection delay, which we quantify. In simulation the martingale needs 20-27% more items than a fixed-sample exact binomial test, which, checked after every item, opens 15 times its error budget at the ceiling; pooling opens topics from about 20 items. Three pre-registered pilots: on Blocksworld a weak engine let the gate open one step type (2 of 50 unchecked steps there wrong); with reasoning it ran 692 steps unchecked without error at 41% of the checks. In ScienceWorld, with a mixed profile and each check costing an action, the gate completed 63.4% of 309 paired episodes, against 59.2% for checking until the budget ran out (p=0.0012) and 54.7% for random checking at near-equal spend; 4 of 464 unchecked steps were wrong. Files: the preprint; the gate simulation script; the script of the checks of Section 4; the script that recomputes the third pilot's numbers from its raw logs (the logs are available from the first author on request). Requires Python 3 with numpy and scipy.
Authors
- Laura La Gattuta
- Alvise Benetazzo (ORCID: https://orcid.org/0000-0002-9535-4922)
- Alessio Langiu (ORCID: https://orcid.org/0000-0002-5706-7500)
- Giovanni Pilato (ORCID: https://orcid.org/0000-0002-6254-2249)
Institutions
- Institute for High Performance Computing and Networking (IT)
- Istituto di Scienze Marine del Consiglio Nazionale delle Ricerche (IT)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-28
- DOI
- https://doi.org/10.5281/zenodo.23720803
- Primary Topic
- AI-based Problem Solving and Planning
- Type
- preprint