Ergodicity in reinforcement learning
In reinforcement learning, we typically aim to optimize the expected value of the sum of rewards an agent collects over a trajectory. However, if the process generating these rewards is non-ergodic, the expected value, i.e. the average over infinitely many trajectories with a given policy, is uninformative for the average over a single, but infinitely long trajectory. Thus, if we care about how the individual agent performs during deployment, the expected value is not a good optimization objective. In this paper, we discuss the impact of non-ergodic reward processes on reinforcement learning agents through an instructive example, relate the notion of ergodic reward processes to more widely used notions of ergodic Markov chains, and present existing solutions that optimize long-term performance of individual trajectories under non-ergodic reward dynamics. This article is part of the theme issue ‘Ergodicity and ergodicity breaking in the sciences’.
Authors
- Arsenii Mustafin
- Dominik Baumann (ORCID: https://orcid.org/0000-0001-7340-2180)
- Vincent Ginis (ORCID: https://orcid.org/0000-0003-0063-9608)
- Erfaun Noorani (ORCID: https://orcid.org/0000-0002-5171-122X)
- Thomas B. Schön (ORCID: https://orcid.org/0000-0001-5183-234X)
- Bert Verbruggen (ORCID: https://orcid.org/0000-0001-9776-2420)
- Arne Vanhoyweghen (ORCID: https://orcid.org/0000-0003-0103-4715)
- Xinyi Sheng
Institutions
- Uppsala University (SE)
- Vrije Universiteit Brussel (BE)
- Harvard University (US)
- MIT Lincoln Laboratory (US)
- Aalto University (FI)
Publication Details
- Journal
- Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences
- Published
- 2026-11-19
- DOI
- https://doi.org/10.1098/rsta.2025.0156
- Citations
- 1
- Primary Topic
- Reinforcement Learning in Robotics
- Type
- article
- Field-Weighted Citation Impact
- 0.97
Funders
- Finnish Center for Artificial Intelligence