Reinforcement learning in path-dependent, non-ergodic contexts

Over the past decades, humanity has tried to automate decision-making processes, often by optimizing a specific value function. But how do you make a decision when significant risk is involved? Central to decision-making under uncertainty often lies the assumption of ergodicity, which implies that temporal evolution can be predicted from ensemble averages. In this article, we study what happens to a reinforcement learning (RL) agent when ergodicity is broken in a decision-making context. First, we show that in a traditional implementation, RL algorithms optimize towards an expected value of a stochastic process, having detrimental consequences for the temporal evolution of the agents in such a setting. Second, we propose a general mechanism that allows RL agents to capture the effects of ergodicity breaking and path dependencies, enabling them to converge in a more optimal strategy in terms of temporal evolution. Finally, we implement this training mechanism in a variety of RL experiments. These experiments show that our proposed method works for ergodic dynamics and non-ergodic contexts. We further apply our temporal training method to a simplified portfolio selection problem, and we show that our method allows an RL agent to approximate the Kelly criterion without transforming the value function.