Improving Visual Reinforcement Learning via Spatial-Temporal Data Augmentation

Sample efficiency has been a challenging problem in visual reinforcement learning, where agents not only learn polices but also extract meaningful state representations for decision making from images. Reinforcement Learning methods that adopt data augmentation have significantly improved sample efficiency. However, prior studies have primarily focused on spatial augmentation, which enhances data diversity but insufficiently exploits temporal information, such as historical states which is critical for time-series-dependent reinforcement learning tasks. To address this gap, we introduce a Spatial-Temporal Data Augmentation (STDA) framework designed to enhance the agent's ability to learn task-relevant state representations by effectively leveraging historical state information. We evaluate our approach using the DeepMind Control Suite, and empirical results demonstrate that STDA significantly improves performance, surpassing state-of-the-art methods across a range of tasks.