DC-Debates: Difficulty-Calibrated Debate Transcripts as Stimuli for Scalable Oversight Research with Human Participants

Approaches to scalable oversight involving human judges, such as AI safety via debate, will be more likely to achieve success if systematic errors in judgment can be minimized. Research on approaches to reducing this error, such as different ways of aggregating participant judgments or manipulating instructions to human participants, is more likely to generalise well if conducted using stimuli analogous to the kinds of stimuli that we expect strong trained AI debaters to generate. This requires well-designed transcripts that are difficult enough to leave room for interventions to improve judge accuracy, but tractable enough that an attentive judge has some hope of arriving at the correct answer. We describe the construction of 24 debate transcripts spanning 12 questions in three domains (astrophysics research, brainteasers, and reward-hacking evaluations) that assume an underlying recursive protocol, with section-level balancing across 11 stylistic and rhetorical dimensions. We report a preregistered between-subjects study with human judges (N = 1,282) which estimates per-transcript posterior difficulty on the logit scale and partitions the 24 transcripts into a four-tier ordinal difficulty matrix. Our results show that our stimuli span a range of difficulty levels, providing a calibrated stimulus pool for subsequent research. We also report several failure modes in which honest debaters did not bring judges to the correct answer despite faithful argumentation, including a recursive-protocol-specific failure in which the dishonest debater steers recursion onto the less important of two subclaims. All transcripts, judge instructions, and analysis code are released as supplementary materials to facilitate follow-up work.

Authors

Publication Details

Journal
PsyArXiv (OSF Preprints)
Published
2026-11-20
DOI
https://doi.org/10.31234/osf.io/f56sc_v2
Primary Topic
Ethics and Social Impacts of AI
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

DC-Debates: Difficulty-Calibrated Debate Transcripts as Stimuli for Scalable Oversight Research with Human Participants

Gabriel L. Recchia, Chatrik Singh Mangat, Sanika Khadkikar, Suhasini Satish Rao
PsyArXiv (OSF Preprints)
Ethics and Social Impacts of AI
preprint

DC-Debates: Difficulty-Calibrated Debate Transcripts as Stimuli for Scalable Oversight Research with Human Participants

Gabriel L. Recchia, Chatrik Singh Mangat, Sanika Khadkikar, Suhasini Satish Rao
preprint en

Abstract

Approaches to scalable oversight involving human judges, such as AI safety via debate, will be more likely to achieve success if systematic errors in judgment can be minimized. Research on approaches to reducing this error, such as different ways of aggregating participant judgments or manipulating instructions to human participants, is more likely to generalise well if conducted using stimuli analogous to the kinds of stimuli that we expect strong trained AI debaters to generate. This requires well-designed transcripts that are difficult enough to leave room for interventions to improve judge accuracy, but tractable enough that an attentive judge has some hope of arriving at the correct answer. We describe the construction of 24 debate transcripts spanning 12 questions in three domains (astrophysics research, brainteasers, and reward-hacking evaluations) that assume an underlying recursive protocol, with section-level balancing across 11 stylistic and rhetorical dimensions. We report a preregistered between-subjects study with human judges (N = 1,282) which estimates per-transcript posterior difficulty on the logit scale and partitions the 24 transcripts into a four-tier ordinal difficulty matrix. Our results show that our stimuli span a range of difficulty levels, providing a calibrated stimulus pool for subsequent research. We also report several failure modes in which honest debaters did not bring judges to the correct answer despite faithful argumentation, including a recursive-protocol-specific failure in which the dishonest debater steers recursion onto the less important of two subclaims. All transcripts, judge instructions, and analysis code are released as supplementary materials to facilitate follow-up work.

PsyArXiv (OSF Preprints)
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon