Controlling long-horizon behavior in reinforcement learning

A central concern of Artificial Intelligence research is constructing agents that can interact with an unknown environment to achieve goals and learn from their experiences. Although several approaches to this problem have been proposed, reinforcement learning (RL) stands out for its generality. Nearly anything can be framed as an RL problem. RL algorithms are used in robotics, to fine-tune language models, to make financial trades, to run industrial plants, and to prove mathematical theorems. Excitingly, this means that successful RL algorithms can have an impact across an incredibly wide range of fields. However, this generality is in many ways a double-edged sword -- because the problem is so general, designing algorithms that are efficient across this staggering array of domains can be incredibly challenging. For several reasons, RL algorithms typically struggle with problems that have long time horizons. Firstly, exploration tends to become exponentially harder as time horizons grow because the number of possible action sequences is exponential in the length of the sequence. Another problem is that ensuring stability is more difficult in problems with longer time horizons. Model-based methods can be chaotic, causing their error to grow exponentially over time, and imitation learning-based methods can suffer from distribution shift that causes their error to grow. Addressing these challenges is necessary for reinforcement learning to become a practical solution to a wider range of problems. In this thesis, I expand on a set of mathematical tools for addressing these issues. Specifically, I argue that the state occupancy measure can be used to address many of these problems and make them tractable, especially when combined with regularization strategies from the online learning and bandit literature.