Advances in long-horizon planning via transformer and diffusion world models

Model-based deep reinforcement learning (MBRL) leverages learned models of environment dynamics to facilitate more efficient planning and decision-making. Recent advancements have improved model accuracy, uncertainty estimation, and planning in latent spaces, enabling MBRL to perform competitively in complex control tasks. However, when applied to long-horizon problems, MBRL still faces critical challenges, including compounding model errors, difficulties in credit assignment under sparse rewards, and limited generalization beyond the training distribution. This dissertation addresses these challenges through innovations in both architectural design and modeling strategies, and is organized into three main parts. In the first part, we show that employing Transformer-based world models significantly improves long-horizon prediction accuracy by effectively capturing long-range temporal dependencies. However, the use of autoregressive generation introduces compounding errors over time. In the second part, we reformulate reinforcement learning as a sequence modeling problem and demonstrate that a hierarchical diffusion planner can achieve strong performance in long-horizon planning tasks. To address the limitations of diffusion planners in accurately estimating value, we further show that integrating value learning at the lower level enables the hierarchical diffusion planner to excel in both sparse-reward and dense-reward environments. Finally, in the third part, we tackle a fundamental limitation of sequence modeling approaches: their inability to generalize beyond the training distribution. As a distribution modeling technique, the diffusion planner is constrained by the data it has seen and struggles to generate valid plans when start-goal connections are absent. To overcome this, we propose a stitch-and-plan framework that enables generalization beyond the training data by composing complete trajectories from feasible sub-trajectories, effectively stitching together partial plans to form coherent solutions.