Scalable Oversight for Advanced AI: Recursive Reward Modeling and Debate

Reinforcement learning from human feedback trains language models against a reward model fitted to human comparisons, and it works only as long as the people giving feedback can tell a good output from a bad one. Scalable oversight is the problem of supervising systems whose outputs exceed that competence. This article reviews the two oldest proposals for the problem, recursive reward modelling with its close relative iterated amplification, and debate between competing models before a human judge, and places them beside two newer lines of work, weak-to-strong generalization and model-written critique. The review first sets out why direct feedback fails under optimization pressure, drawing on published evidence of reward hacking, reward model overoptimization, sycophancy and unintended persuasion. It then examines the assumptions each oversight method needs, using simple cost and error formulas to show what decomposition and adversarial argument gain and where their guarantees stop. The empirical record on debate is reported as it stands, with clear gains for multi-turn debate in reading comprehension tasks where the judge cannot see the source passage and small or mixed effects elsewhere. A comparative framework is proposed that links each method to its required assumption, its characteristic failure mode and a diagnostic test, together with an evaluation protocol built on the sandwiching design. The framework is analytical and has not yet been tested empirically.

Authors

Publication Details

Journal
Eduschool Journal of Artificial Intelligence Research (EJAIR)
Published
2026-11-20
DOI
https://doi.org/10.63090/ejair/3139.7336.0012
Primary Topic
Ethics and Social Impacts of AI
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Scalable Oversight for Advanced AI: Recursive Reward Modeling and Debate

Raji N
1 links
Eduschool Journal of Artificial Intelligence Research (EJAIR)
Ethics and Social Impacts of AI
article

Scalable Oversight for Advanced AI: Recursive Reward Modeling and Debate

Raji N
article en

Abstract

Reinforcement learning from human feedback trains language models against a reward model fitted to human comparisons, and it works only as long as the people giving feedback can tell a good output from a bad one. Scalable oversight is the problem of supervising systems whose outputs exceed that competence. This article reviews the two oldest proposals for the problem, recursive reward modelling with its close relative iterated amplification, and debate between competing models before a human judge, and places them beside two newer lines of work, weak-to-strong generalization and model-written critique. The review first sets out why direct feedback fails under optimization pressure, drawing on published evidence of reward hacking, reward model overoptimization, sycophancy and unintended persuasion. It then examines the assumptions each oversight method needs, using simple cost and error formulas to show what decomposition and adversarial argument gain and where their guarantees stop. The empirical record on debate is reported as it stands, with clear gains for multi-turn debate in reading comprehension tasks where the judge cannot see the source passage and small or mixed effects elsewhere. A comparative framework is proposed that links each method to its required assumption, its characteristic failure mode and a diagnostic test, together with an evaluation protocol built on the sandwiching design. The framework is analytical and has not yet been tested empirically.

Eduschool Journal of Artificial Intelligence Research (EJAIR)Vol. 1(3)
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon