Hierarchical Quasimetric Reinforcement Learning

Offline Goal-Conditioned Reinforcement Learning (Offline GCRL) has emerged as a promising paradigm for solving multi-task RL problems by leveraging static datasets and goal-conditioned value functions. However, traditional approaches like Quasimetric Reinforcement Learning (QRL) struggle with scalability in high-dimensional environments due to state space sparsity and optimization challenges. To address this, we propose a hierarchical QRL framework that integrates contrastive learning and imitation learning. The upper-level model constructs a reduced goal space using contrastive learning and learns a goal-conditioned value function to estimate distances between goals, while the lower-level model employs imitation learning to execute short-horizon policies. Our approach demonstrates superior performance in offline RL benchmarks, offering better generalization and adaptability for complex, high-dimensional tasks.