When May an Agent Act Alone? An Anytime-Valid Gate on a Measured Competence Profile

Language-model agents decide, at every step of a long task, whether to act alone or to have the step checked first. We base this decision on a measured record, not on the model's self-assessment: a competence profile holding, per topic, how often an external verifier found the engine right, wrong, or could not check it. A gate lets steps on a topic proceed unchecked only after a likelihood-ratio test martingale rejects "the contradiction rate on this topic is at least the ceiling". The test is classical; the protocol around it is the contribution: pooling over complete rounds of an abstract type opens sparse topics earlier, a seeded random audit tests the unchecked steps and closes a topic whose rate rises, and a block rule handles dependence inside a task. Proposition 1 bounds, with probability at least 1-delta-delta_eta, for every topic at once, at every time of a run whose rates do not drift and under the planner's own stopping rule, the probability that a checkable result executed unchecked is wrong: at most epsilon plus one stated term per broken assumption (predictor miss, distribution shift, within-type dispersion). Under drift the bound becomes a detection delay, which we quantify. In simulation the martingale needs 20-27% more items than a fixed-sample exact binomial test, which, checked after every item, opens 15 times its error budget at the ceiling; pooling opens topics from about 20 items. Three pre-registered pilots: on Blocksworld a weak engine let the gate open one step type (2 of 50 unchecked steps there wrong); with reasoning it ran 692 steps unchecked without error at 41% of the checks. In ScienceWorld, with a mixed profile and each check costing an action, the gate completed 63.4% of 309 paired episodes, against 59.2% for checking until the budget ran out (p=0.0012) and 54.7% for random checking at near-equal spend; 4 of 464 unchecked steps were wrong. Files: the preprint; the gate simulation script; the script of the checks of Section 4; the script that recomputes the third pilot's numbers from its raw logs (the logs are available from the first author on request). Requires Python 3 with numpy and scipy.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-28
DOI
https://doi.org/10.5281/zenodo.23720803
Primary Topic
AI-based Problem Solving and Planning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

When May an Agent Act Alone? An Anytime-Valid Gate on a Measured Competence Profile

Laura La Gattuta, Alvise Benetazzo, Alessio Langiu, Giovanni Pilato
2 links
Zenodo (CERN European Organization for Nuclear Research)
AI-based Problem Solving and Planning
preprint

When May an Agent Act Alone? An Anytime-Valid Gate on a Measured Competence Profile

Laura La Gattuta, Alvise Benetazzo, Alessio Langiu, Giovanni Pilato
preprint en

Abstract

Language-model agents decide, at every step of a long task, whether to act alone or to have the step checked first. We base this decision on a measured record, not on the model's self-assessment: a competence profile holding, per topic, how often an external verifier found the engine right, wrong, or could not check it. A gate lets steps on a topic proceed unchecked only after a likelihood-ratio test martingale rejects "the contradiction rate on this topic is at least the ceiling". The test is classical; the protocol around it is the contribution: pooling over complete rounds of an abstract type opens sparse topics earlier, a seeded random audit tests the unchecked steps and closes a topic whose rate rises, and a block rule handles dependence inside a task. Proposition 1 bounds, with probability at least 1-delta-delta_eta, for every topic at once, at every time of a run whose rates do not drift and under the planner's own stopping rule, the probability that a checkable result executed unchecked is wrong: at most epsilon plus one stated term per broken assumption (predictor miss, distribution shift, within-type dispersion). Under drift the bound becomes a detection delay, which we quantify. In simulation the martingale needs 20-27% more items than a fixed-sample exact binomial test, which, checked after every item, opens 15 times its error budget at the ceiling; pooling opens topics from about 20 items. Three pre-registered pilots: on Blocksworld a weak engine let the gate open one step type (2 of 50 unchecked steps there wrong); with reasoning it ran 692 steps unchecked without error at 41% of the checks. In ScienceWorld, with a mixed profile and each check costing an action, the gate completed 63.4% of 309 paired episodes, against 59.2% for checking until the budget ran out (p=0.0012) and 54.7% for random checking at near-equal spend; 4 of 464 unchecked steps were wrong. Files: the preprint; the gate simulation script; the script of the checks of Section 4; the script that recomputes the third pilot's numbers from its raw logs (the logs are available from the first author on request). Requires Python 3 with numpy and scipy.

Zenodo (CERN European Organization for Nuclear Research)
Institute for High Performance Computing and Networking (IT), Istituto di Scienze Marine del Consiglio Nazionale delle Ricerche (IT)
AI-based Problem Solving and Planning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon