Attempted versus Happened: Checking an Agent's Authority Before It Acts on τ²-bench

Benchmarks for customer-service agents score whether the customer ended up with the right outcome. They do not ask whether each change the agent made was one that customer could authorize. This note adds that question to τ²-bench's retail and airline domains with a check that sits in front of every tool call that would change records and compares it with a signed list of what the customer in the conversation may authorize. The check can observe, recording what it would refuse, or enforce, refusing before the call runs. Without calling any language model, it blocks none of the benchmark's 164 reference solutions, refuses all 1,223 planted out-of-authority actions, and changes nothing when only observing. Replaying 5,826 published leaderboard conversations, it finds 50 changes that went through without the customer's authority. 40 of them breach the policy's own text, and 33 of those were in conversations the benchmark scored as passes. Every decision carries a receipt that can be verified offline.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-16
DOI
https://doi.org/10.5281/zenodo.23423056
Primary Topic
Access Control and Trust
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Attempted versus Happened: Checking an Agent's Authority Before It Acts on τ²-bench

Inam Haq
Zenodo (CERN European Organization for Nuclear Research)
Access Control and Trust
preprint

Attempted versus Happened: Checking an Agent's Authority Before It Acts on τ²-bench

Inam Haq
preprint en

Abstract

Benchmarks for customer-service agents score whether the customer ended up with the right outcome. They do not ask whether each change the agent made was one that customer could authorize. This note adds that question to τ²-bench's retail and airline domains with a check that sits in front of every tool call that would change records and compares it with a signed list of what the customer in the conversation may authorize. The check can observe, recording what it would refuse, or enforce, refusing before the call runs. Without calling any language model, it blocks none of the benchmark's 164 reference solutions, refuses all 1,223 planted out-of-authority actions, and changes nothing when only observing. Replaying 5,826 published leaderboard conversations, it finds 50 changes that went through without the customer's authority. 40 of them breach the policy's own text, and 33 of those were in conversations the benchmark scored as passes. Every decision carries a receipt that can be verified offline.

Zenodo (CERN European Organization for Nuclear Research)
Peace, justice, and strong institutions
Access Control and Trust
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.