Per-Attack Blocking Hides Catalog Vulnerability: A Weakest-Link Analysis of Tool Manifests

Runtime controls for tool-using agents depend on manifests that state what each tool may read and where its output may go. Evaluations of these controls, including ChainCaps, report per-attack blocking: the share of attacks a set of manifests stops. We show that this is the wrong unit. A catalog is only as safe as its weakest source policy: if M independently authored policies are each wrong with probability p, the chance that the catalog contains an accepted exfiltration path is 1 - (1 - p)^M, while per-attack blocking stays at 1 - p. Three lines of evidence agree. In synthetic catalogs run through two enforcement engines, ChainCaps and OpenAPPA, per-attack blocking holds at 92% while catalog vulnerability rises from 22% at 10 tools to 99% at 200, and composition provides no protective effect. A fifteen-model authoring pilot shows that real errors cluster by author, so a 200-tool catalog behaves like about 5 independent decisions rather than 60 (95% interval 3.4 to 9.2). On 114 real MCP tools with expert-written rules, the models' own manifests block 89% of attacks yet leave 9 of 15 catalogs open, as the author-effect model predicts and independence does not. Evaluations should report catalog vulnerability, the sink coverage of their tests, and an effective number of independent decisions alongside per-attack blocking. Accepted at the Third Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD), NeurIPS 2026.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24241155
Primary Topic
Security and Verification in Computing
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Per-Attack Blocking Hides Catalog Vulnerability: A Weakest-Link Analysis of Tool Manifests

Karthik Karunanithi
Zenodo (CERN European Organization for Nuclear Research)
Security and Verification in Computing
preprint

Per-Attack Blocking Hides Catalog Vulnerability: A Weakest-Link Analysis of Tool Manifests

Karthik Karunanithi
preprint en

Abstract

Runtime controls for tool-using agents depend on manifests that state what each tool may read and where its output may go. Evaluations of these controls, including ChainCaps, report per-attack blocking: the share of attacks a set of manifests stops. We show that this is the wrong unit. A catalog is only as safe as its weakest source policy: if M independently authored policies are each wrong with probability p, the chance that the catalog contains an accepted exfiltration path is 1 - (1 - p)^M, while per-attack blocking stays at 1 - p. Three lines of evidence agree. In synthetic catalogs run through two enforcement engines, ChainCaps and OpenAPPA, per-attack blocking holds at 92% while catalog vulnerability rises from 22% at 10 tools to 99% at 200, and composition provides no protective effect. A fifteen-model authoring pilot shows that real errors cluster by author, so a 200-tool catalog behaves like about 5 independent decisions rather than 60 (95% interval 3.4 to 9.2). On 114 real MCP tools with expert-written rules, the models' own manifests block 89% of attacks yet leave 9 of 15 catalogs open, as the author-effect model predicts and independence does not. Evaluations should report catalog vulnerability, the sink coverage of their tests, and an effective number of independent decisions alongside per-attack blocking. Accepted at the Third Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD), NeurIPS 2026.

Zenodo (CERN European Organization for Nuclear Research)
Security and Verification in Computing
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Per-Attack Blocking Hides Catalog Vulnerability: A Weakest-Link Analysis of Tool Manifests — Karthik Karunanithi · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS