Do AI Agents Stop Before Acting? A Frozen Benchmark of Lookalike Domains, Poisoned Tools, Revoked Receipts and Stale Answers, With and Without an External Verification Signal

An AI agent that acts on the web for a person must sometimes decide not to act: the link it was handed is a lookalike of a known brand, the tool server it is about to use carries injected instructions, the receipt that authorised a purchase has been revoked, the cached answer it relies on has expired, the site's published rules forbid the fetch, or the page it is about to cite is not what it claims to be. We present a small frozen benchmark of 39 such situations (20 hazards, 19 matched controls) across six families, and run four open-weight or low-cost language models through each case twice: once with only what the agent could observe for itself, and once with the same evidence plus a signed answer from an external verification service. Alone, the models stopped on 32 of 80 hazard trials (40%, 95% CI 30–51%); with the signal, 79 of 80 (99%, CI 93–100%). Usefulness on controls rose from 78% to 89%. The gain is uneven by family: models stopped on every lookalike domain unaided, but on only 1 of 12 robots.txt refusals and 0 of 12 identity mismatches. We publish the suite, the prompt template, the scoring rule, every result and the signatures over them so that the run can be repeated and challenged. We state the limits plainly: the cases were chosen by the operator of the verification service, the signal in the second arm is correct by construction, so the second arm measures whether a model heeds a correct signal rather than how often the service is right; and each case was run once per model. Source, data and code: https://github.com/emmanuelorta/crawlcheck-core/tree/2770804e4ff0372e070c6fcfaadcfdaef9727991/paper.