Reviewed Before It Runs: Automated Review of an Architect's Authority
AI agents are granted authority in two layers: a boundary written in advance for a whole deployment, and a decision on each action as it arrives, made by a person or, increasingly, by a classifier model. Neither states what one task needs before it runs. We study an alternative, in which each task’s authority is written down before anything runs and reviewed there. From a task in natural language, an architect model writes a signal graph, a dataflow graph in which every node declares the capabilities it holds, and every grant quotes the words of the task that need it. Before anything runs, a formal review checks the graph, and a reviewer model, here the architect’s own model in a separate call, reviews it. Because each grant is an artefact, it can be held to least-authority labels registered before the run, so whether review catches a widened grant can be counted. On AgentDojo’s banking suite, with one model (claude-haiku-5-5) as both architect and reviewer, we registered one test: whether a pretext in the task, claiming that wider access had been approved, raises the rate of widenings both reviewers miss. They came in 29 of 160 trials with the pretext (18.1%) and 6 of 80 without it (7.5%): one-sided Fisher exact p = 0.019, against a registered 𝛼 = 0.01. That is below the conventional 0.05 but above the threshold the registration set, so by its registration the test is not supported. Post hoc, the rise is mostly in whether the architect wrote a graph at all: compared within each task, p = 0.0033 per trial but 0.19 per graph (Section 5.1). The reviewer caught all 10 widenings planted to test it before use, but only 17 of the 76 trials in which the architect had widened a grant beyond its label; re-run post hoc on the same graphs, a larger model of the same family caught 22 of the 76, and flagged 6 clean trials against 3. The link check, which requires each grant to quote the task, caught none. The architect’s graphs scored utility on 7/16 tasks (3/11 beyond tasks that pass when nothing is done), against 13/16 (8/11) for the same model undefended, and attack success was zero for both, which shows no defence here.
Authors
- Yoni Lavi (ORCID: https://orcid.org/0000-0002-6164-1499)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-21
- DOI
- https://doi.org/10.5281/zenodo.24422565
- Primary Topic
- Access Control and Trust
- Type
- preprint