When Local Credit Assignment Breaks: Exact Diagnostics of Additive and Pairwise Reward Projection

Abstract Average reward reconstruction error is an incomplete measure of the decision quality of a factored cooperative model. We provide a small, fully reproducible diagnostic of this established issue, rather than a new multi-agent reinforcement learning algorithm. In a binary one-step game with a rare joint synergy, the exact uniform least-squares additive projection has mean squared error 2⁻ⁿ − (n + 1)4⁻ⁿ but coordination regret 1 − 3n2⁻ⁿ. Adding any set of pairwise factors reduces reconstruction error without changing the wrong maximizing action. Exhaustive evaluation at up to 16 agents confirms the formulas; 800 independent sampled fits illustrate that observing the synergistic action need not repair the population projection objective. A privileged twofold weight on the known optimal action reverses the policy while increasing uniform reconstruction error. At 16 agents, additive MSE is 1.52548 × 10⁻⁵ and regret is 0.999268. However, MSE divided by reward variance is 0.999939, exposing the apparently small absolute error as a sparse-reward averaging effect. The package supplies proofs, controlled ablations, an established matrix-game check, raw results, and tests. It does not reproduce LOMAQ or establish a new failure of neural value-decomposition algorithms.

Authors

Publication Details

Journal
Research Square
Published
2026-11-10
DOI
https://doi.org/10.21203/rs.3.rs-11657377/v1
Primary Topic
Reinforcement Learning in Robotics
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

When Local Credit Assignment Breaks: Exact Diagnostics of Additive and Pairwise Reward Projection

Krishna Varun Kasamneni
Research Square
Reinforcement Learning in Robotics
preprint

When Local Credit Assignment Breaks: Exact Diagnostics of Additive and Pairwise Reward Projection

Krishna Varun Kasamneni
preprint en

Abstract

Abstract Average reward reconstruction error is an incomplete measure of the decision quality of a factored cooperative model. We provide a small, fully reproducible diagnostic of this established issue, rather than a new multi-agent reinforcement learning algorithm. In a binary one-step game with a rare joint synergy, the exact uniform least-squares additive projection has mean squared error 2⁻ⁿ − (n + 1)4⁻ⁿ but coordination regret 1 − 3n2⁻ⁿ. Adding any set of pairwise factors reduces reconstruction error without changing the wrong maximizing action. Exhaustive evaluation at up to 16 agents confirms the formulas; 800 independent sampled fits illustrate that observing the synergistic action need not repair the population projection objective. A privileged twofold weight on the known optimal action reverses the policy while increasing uniform reconstruction error. At 16 agents, additive MSE is 1.52548 × 10⁻⁵ and regret is 0.999268. However, MSE divided by reward variance is 0.999939, exposing the apparently small absolute error as a sparse-reward averaging effect. The package supplies proofs, controlled ablations, an established matrix-game check, raw results, and tests. It does not reproduce LOMAQ or establish a new failure of neural value-decomposition algorithms.

Research Square
Reinforcement Learning in Robotics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon