Ergodicity in reinforcement learning

In reinforcement learning, we typically aim to optimize the expected value of the sum of rewards an agent collects over a trajectory. However, if the process generating these rewards is non-ergodic, the expected value, i.e. the average over infinitely many trajectories with a given policy, is uninformative for the average over a single, but infinitely long trajectory. Thus, if we care about how the individual agent performs during deployment, the expected value is not a good optimization objective. In this paper, we discuss the impact of non-ergodic reward processes on reinforcement learning agents through an instructive example, relate the notion of ergodic reward processes to more widely used notions of ergodic Markov chains, and present existing solutions that optimize long-term performance of individual trajectories under non-ergodic reward dynamics. This article is part of the theme issue ‘Ergodicity and ergodicity breaking in the sciences’.

Authors

Institutions

Publication Details

Journal
Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences
Published
2026-11-19
DOI
https://doi.org/10.1098/rsta.2025.0156
Citations
1
Primary Topic
Reinforcement Learning in Robotics
Type
article
Field-Weighted Citation Impact
0.97

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Ergodicity in reinforcement learning

Arsenii Mustafin, Dominik Baumann, Vincent Ginis, Erfaun Noorani et al.
1 citations
Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences
Reinforcement Learning in Robotics
0.97
article

Ergodicity in reinforcement learning

Arsenii Mustafin, Dominik Baumann, Vincent Ginis, Erfaun Noorani, Thomas B. Schön, Bert Verbruggen, Arne Vanhoyweghen, Xinyi Sheng
article en
1 citations

Abstract

In reinforcement learning, we typically aim to optimize the expected value of the sum of rewards an agent collects over a trajectory. However, if the process generating these rewards is non-ergodic, the expected value, i.e. the average over infinitely many trajectories with a given policy, is uninformative for the average over a single, but infinitely long trajectory. Thus, if we care about how the individual agent performs during deployment, the expected value is not a good optimization objective. In this paper, we discuss the impact of non-ergodic reward processes on reinforcement learning agents through an instructive example, relate the notion of ergodic reward processes to more widely used notions of ergodic Markov chains, and present existing solutions that optimize long-term performance of individual trajectories under non-ergodic reward dynamics. This article is part of the theme issue ‘Ergodicity and ergodicity breaking in the sciences’.

Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering SciencesVol. 384(2334)
Uppsala University (SE), Vrije Universiteit Brussel (BE), Harvard University (US), MIT Lincoln Laboratory (US), Aalto University (FI)
Finnish Center for Artificial Intelligence
Openalex Percentile: Top 23%
Reinforcement Learning in Robotics
0.97
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon