Model-agnostic solutions for deep reinforcement learning in non-ergodic contexts

Reinforcement learning (RL) remains a central optimization framework in machine learning. Although RL agents can converge to optimal solutions, the definition of ‘optimality’ depends on the environment's statistical properties. The Bellman equation, central to most RL algorithms, is formulated in terms of expected values of future rewards. However, when ergodicity is broken, long-term outcomes depend on the specific trajectory rather than on the ensemble average. In such settings, the ensemble average diverges from the time-average growth experienced by individual agents, with expected-value formulations yielding systematically suboptimal policies. Prior studies demonstrated that traditional RL architectures fail to recover the true optimum in non-ergodic environments. We extend this analysis to deep RL implementations and show that these, too, produce suboptimal policies under non-ergodic dynamics. We show that this limitation can be addressed by treating non-ergodic decision problems as finite trajectory-exposure problems rather than as isolated one-step optimization tasks. By exposing the agent to compounded trajectories during training, the learned value estimates become more consistent with the process's intrinsic time growth. This improvement does not require altering the environmental feedback, such as reward transformations or modified objective functions, but arises naturally from the agent's exposure to temporal trajectories. Our results contribute to the growing body of research on RL methods for non-ergodic systems. This article is part of the theme issue ‘Ergodicity and ergodicity breaking in the sciences’.

Authors

Institutions

Publication Details

Journal
Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences
Published
2026-11-19
DOI
https://doi.org/10.1098/rsta.2025.0154
Citations
2
Primary Topic
Reinforcement Learning in Robotics
Type
article
Field-Weighted Citation Impact
1.94
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Model-agnostic solutions for deep reinforcement learning in non-ergodic contexts

Bert Verbruggen, Arne Vanhoyweghen, Vincent Ginis
2 citations
Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences
Reinforcement Learning in Robotics
1.94
article

Model-agnostic solutions for deep reinforcement learning in non-ergodic contexts

Bert Verbruggen, Arne Vanhoyweghen, Vincent Ginis
article en
2 citations

Abstract

Reinforcement learning (RL) remains a central optimization framework in machine learning. Although RL agents can converge to optimal solutions, the definition of ‘optimality’ depends on the environment's statistical properties. The Bellman equation, central to most RL algorithms, is formulated in terms of expected values of future rewards. However, when ergodicity is broken, long-term outcomes depend on the specific trajectory rather than on the ensemble average. In such settings, the ensemble average diverges from the time-average growth experienced by individual agents, with expected-value formulations yielding systematically suboptimal policies. Prior studies demonstrated that traditional RL architectures fail to recover the true optimum in non-ergodic environments. We extend this analysis to deep RL implementations and show that these, too, produce suboptimal policies under non-ergodic dynamics. We show that this limitation can be addressed by treating non-ergodic decision problems as finite trajectory-exposure problems rather than as isolated one-step optimization tasks. By exposing the agent to compounded trajectories during training, the learned value estimates become more consistent with the process's intrinsic time growth. This improvement does not require altering the environmental feedback, such as reward transformations or modified objective functions, but arises naturally from the agent's exposure to temporal trajectories. Our results contribute to the growing body of research on RL methods for non-ergodic systems. This article is part of the theme issue ‘Ergodicity and ergodicity breaking in the sciences’.

Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering SciencesVol. 384(2334)
Vrije Universiteit Brussel (BE), Harvard University (US)
Openalex Percentile: Top 13%
Reinforcement Learning in Robotics
1.94
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon