Introducing an Adapter into the Pre-trained Encoder for Visual Reinforcement Learning

Recent unsupervised pre-training techniques have proven effective in the visual reinforcement learning domain by generating valuable representations for various downstream tasks. Existing methods aim to obtain a robust representation by completely freezing the parameters of the encoder. How-ever, the visual representations learned in this way are not necessarily fully applicable to specific downstream tasks. In this paper, we introduce a novel method designed to enhance the ability of models to learn useful visual representations. Specifically, we acquire a robust representation via introducing data augmentation to diversify the appearance of in-domain observations. Subsequently, we utilize a pre-trained encoder to extract visual representations, but with minor modifications to the structure of the model. In the downstream task, we freeze the model and introduce Adapter into the Pre-trained Encoder (AiPE) for efficient parameter fine-tuning. The visual representations extracted by this approach can be better adapted to specific downstream tasks. We conduct extensive experiments on the DMControl Generalization Benchmark and DMControl Manipulation Tasks to validate the effectiveness of our method. Empirical results indicate that our approach enhances sample efficiency and markedly surpasses previous leading methods in generalization performance.