The Construct Validity of White-Box Collusion Monitors: Matched Evidence from LLM Committees

Benchmarks for multi-agent collusion detection report high AUROC for activation monitors, but high ranking accuracy does not establish specificity for covert coordination when compared groups differ in incentives, instructions, partner knowledge, and generated messages. We study this gap in matched LLM committees with two open-weight models, using prompt and transcript contrasts, context attribution, and activation interventions. Lone agents given a stake without a goal instruction or private channel almost always vote for the fair option. A retrained probe separates secret coalition members from baseline controls with AUROC 1.000, yet flags 96.8–100% of these lone agents. At the same time, the probe consistently ranks secret coalition members above matched lone agents given a secret goal, indicating that poor alarm specificity can coexist with coalition signal in the same monitor. On fixed control transcripts, the secret coalition prompt accounts for 88.1–94.6% of the full score shift. Under context replacement, the agenda block has the largest mean Shapley attribution for shifts in secret coalition members’ decision margins. Across responsive decisions, full-state patches at late layers recover over 90% of the agenda removal effect, whereas patches along the fitted agenda direction recover at most 2.3%. No tested ablation or steering condition meets our joint operational criterion for directional specificity. These results show that ranking accuracy, collusion specificity, and directional control require separate evaluation.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24374176
Primary Topic
Ethics and Social Impacts of AI
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Construct Validity of White-Box Collusion Monitors: Matched Evidence from LLM Committees

Bo Zhao
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
preprint

The Construct Validity of White-Box Collusion Monitors: Matched Evidence from LLM Committees

Bo Zhao
preprint en

Abstract

Benchmarks for multi-agent collusion detection report high AUROC for activation monitors, but high ranking accuracy does not establish specificity for covert coordination when compared groups differ in incentives, instructions, partner knowledge, and generated messages. We study this gap in matched LLM committees with two open-weight models, using prompt and transcript contrasts, context attribution, and activation interventions. Lone agents given a stake without a goal instruction or private channel almost always vote for the fair option. A retrained probe separates secret coalition members from baseline controls with AUROC 1.000, yet flags 96.8–100% of these lone agents. At the same time, the probe consistently ranks secret coalition members above matched lone agents given a secret goal, indicating that poor alarm specificity can coexist with coalition signal in the same monitor. On fixed control transcripts, the secret coalition prompt accounts for 88.1–94.6% of the full score shift. Under context replacement, the agenda block has the largest mean Shapley attribution for shifts in secret coalition members’ decision margins. Across responsive decisions, full-state patches at late layers recover over 90% of the agenda removal effect, whereas patches along the fitted agenda direction recover at most 2.3%. No tested ablation or steering condition meets our joint operational criterion for directional specificity. These results show that ranking accuracy, collusion specificity, and directional control require separate evaluation.

Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon