Local Moment Agreement Does Not Preserve Decision Exponents in Coupled Q-Learning

We study wrong greedy decisions made from the last iterate of constant-step Q-learning at stationarity. Bias, variance and a Gaussian approximation describe the distribution near the optimal values. We ask how much they tell us about rare decision errors. For one learner, the frozen-policy linearization gives the exact error exponent in every finite MDP with deterministic transitions, noisy rewards and synchronous sampling, under the assumptions stated below. This property can fail for the Richardson-Romberg (RR) combination of two learners. Both learners use the true Bellman max and share the same samples. In a bounded one-state example, the real pair and its linearization agree in every fixed polynomial moment up to errors smaller than any power of the step size. Yet the ratio of their RR error probabilities has an exponent above 0.07315. A three-state cycle with independent reward noises gives a bound above 0.0538. Both bounds also hold against a real ordinary learner with the same leading variance. We prove a stationary variational formula and use it to study the effect of the discount factor. With Gaussian noise and step ratio 𝑐 > 1, the RR exponent divided by the exponent of the variance-matched learner tends to 𝑐/(𝑐 + 1) in the one-state model as the discount factor tends to one. On every cycle of fixed length, the limsup is at most this value. For 𝑐 = 2, the loss is one third in the scalar model and at least one third in the cycle limit. We also give a small-discount expansion with a quadratic remainder. These results concern unaveraged outputs and do not challenge existing bias or averaging guarantees for RR.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24188417
Primary Topic
Reinforcement Learning in Robotics
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Local Moment Agreement Does Not Preserve Decision Exponents in Coupled Q-Learning

Yaxin Zheng
Zenodo (CERN European Organization for Nuclear Research)
Reinforcement Learning in Robotics
preprint

Local Moment Agreement Does Not Preserve Decision Exponents in Coupled Q-Learning

Yaxin Zheng
preprint en

Abstract

We study wrong greedy decisions made from the last iterate of constant-step Q-learning at stationarity. Bias, variance and a Gaussian approximation describe the distribution near the optimal values. We ask how much they tell us about rare decision errors. For one learner, the frozen-policy linearization gives the exact error exponent in every finite MDP with deterministic transitions, noisy rewards and synchronous sampling, under the assumptions stated below. This property can fail for the Richardson-Romberg (RR) combination of two learners. Both learners use the true Bellman max and share the same samples. In a bounded one-state example, the real pair and its linearization agree in every fixed polynomial moment up to errors smaller than any power of the step size. Yet the ratio of their RR error probabilities has an exponent above 0.07315. A three-state cycle with independent reward noises gives a bound above 0.0538. Both bounds also hold against a real ordinary learner with the same leading variance. We prove a stationary variational formula and use it to study the effect of the discount factor. With Gaussian noise and step ratio 𝑐 > 1, the RR exponent divided by the exponent of the variance-matched learner tends to 𝑐/(𝑐 + 1) in the one-state model as the discount factor tends to one. On every cycle of fixed length, the limsup is at most this value. For 𝑐 = 2, the loss is one third in the scalar model and at least one third in the cycle limit. We also give a small-discount expansion with a quadratic remainder. These results concern unaveraged outputs and do not challenge existing bias or averaging guarantees for RR.

Zenodo (CERN European Organization for Nuclear Research)
Université Paris-Sud (FR), Université de Versailles Saint-Quentin-en-Yvelines (FR)
Reinforcement Learning in Robotics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon