Attempted versus Happened: Checking an Agent's Authority Before It Acts on τ²-bench
Benchmarks for customer-service agents score whether the customer ended up with the right outcome. They do not ask whether each change the agent made was one that customer could authorize. This note adds that question to τ²-bench's retail and airline domains with a check that sits in front of every tool call that would change records and compares it with a signed list of what the customer in the conversation may authorize. The check can observe, recording what it would refuse, or enforce, refusing before the call runs. Without calling any language model, it blocks none of the benchmark's 164 reference solutions, refuses all 1,223 planted out-of-authority actions, and changes nothing when only observing. Replaying 5,826 published leaderboard conversations, it finds 50 changes that went through without the customer's authority. 40 of them breach the policy's own text, and 33 of those were in conversations the benchmark scored as passes. Every decision carries a receipt that can be verified offline.
Authors
- Inam Haq
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-16
- DOI
- https://doi.org/10.5281/zenodo.23423056
- Primary Topic
- Access Control and Trust
- Type
- preprint