Scalable Oversight for Advanced AI: Recursive Reward Modeling and Debate
Reinforcement learning from human feedback trains language models against a reward model fitted to human comparisons, and it works only as long as the people giving feedback can tell a good output from a bad one. Scalable oversight is the problem of supervising systems whose outputs exceed that competence. This article reviews the two oldest proposals for the problem, recursive reward modelling with its close relative iterated amplification, and debate between competing models before a human judge, and places them beside two newer lines of work, weak-to-strong generalization and model-written critique. The review first sets out why direct feedback fails under optimization pressure, drawing on published evidence of reward hacking, reward model overoptimization, sycophancy and unintended persuasion. It then examines the assumptions each oversight method needs, using simple cost and error formulas to show what decomposition and adversarial argument gain and where their guarantees stop. The empirical record on debate is reported as it stands, with clear gains for multi-turn debate in reading comprehension tasks where the judge cannot see the source passage and small or mixed effects elsewhere. A comparative framework is proposed that links each method to its required assumption, its characteristic failure mode and a diagnostic test, together with an evaluation protocol built on the sandwiching design. The framework is analytical and has not yet been tested empirically.
Authors
- Raji N
Publication Details
- Journal
- Eduschool Journal of Artificial Intelligence Research (EJAIR)
- Published
- 2026-11-20
- DOI
- https://doi.org/10.63090/ejair/3139.7336.0012
- Primary Topic
- Ethics and Social Impacts of AI
- Type
- article
- Field-Weighted Citation Impact
- 0.00