Reinforcement Learning with World Models in the Era of Scale: From Online Imagination to World Action Models

Abstract In World Model Reinforcement Learning (WM-RL), agents learn a parameterized model of environmental dynamics that serves as the basis for planning, imagination-based policy training, or synthetic data generation in decision making tasks. From 2019 to 2026, a drastic phase transition took place within this field. Initially started as an effort to improve efficiency in visual control tasks, WM-RL has become the core of interdisciplinary AI infrastructure. Today, internet-scale video-based generative world foundation models allow the autonomous vehicle pipeline to learn generalist policies from unlabeled human video, facilitating applications in medical therapy and molecular dynamics. To review the entire trajectory of WM-RL research, it is necessary to synthesize key sources that summarize the advances made throughout the years. Three prominent paradigm shifts are observed during this period. These include the shift from online Model-Based Reinforcement Learning (MBRL) to pretrain-first imagination-only policy training; architecture integration from decoupled worlds and policies to unified World Action Models; and the generalization of specific algorithms to physical AI infrastructure for all MBRL applications. Regarding architecture, WM-RL follows several generations of innovations. They include PlaNet, Recurrent State-Space Model (RSSM), the Dreamer family, value-equivalent implicit models, Transformer-based world models, diffusion dynamics, and the abstract-space Joint-Embedding Predictive Architecture (JEPA) predictor. In this review, we examine WM-RL from multiple aspects, ranging from general model-based reinforcement learning, robotic manipulation, navigation, autonomous driving, multi-agent systems, large language model agents, healthcare, and games. The usage of world models varies widely depending on the application areas, while the validated architectures are discussed accordingly.