Abstraction Symbolique et Automatique d’Objectifs en Apprentissage par Renforcement Hiérarchique via Analyse d’Atteignabilité

Hierarchical Reinforcement Learning (HRL) is a paradigm that can be leveraged to automatically learn strategies for long-horizon tasks, which typically involve multiple milestones that must be achieved before the problem is solved. The main idea behind Hierarchical Reinforcement Learning is to break up the difficult task into smaller sub-tasks, that can be more easily approached under in a more constrained aspect.A core challenge in HRL is to identify an ideal decomposition of the long-horizon task in the form of goals that a learning agent will try to achieve. High-dimensional environments and complex dynamics make it particularly difficult for the agent to understand which goals are critical for the task.This thesis explores the concept of learning symbolic goal representations within HRL, inspired from abstractions studied in the field of Formal Methods. We develop a spatial abstraction method that captures reachability relations in the environment's observable space, and provide guarantees on the suboptimality of the agent's learned policy. We also prove that the goal abstraction can be computed through a process of refinement. Furthermore, we implement the reachability-aware goal abstraction with a Hierarchical Reinforcement Learning framework called GARA, creating an agent that can concurrently learn the goal abstraction and policy. We showcase the impact of the goal abstraction in the agent's learning efficiency, transferability and interpretability on a set of low-dimensional navigation tasks. In high-dimensional tasks, the abstract goals that be initially too difficult to achieve before refinement. To remedy this issue, we propose a novel algorithm STAR that leverages the reachability-aware spatial abstraction along with a temporal abstraction mechanism allowing for more flexibility on the difficulty of chosen goals. We empirically demonstrate that STAR outperforms the state of the art on a set of difficult continuous control tasks