Experience Extractor for Adaptive Tradeoff Between Exploration and Exploitation in Reinforcement Learning

In Reinforcement Learning (RL), the agent cannot distinguish between exploratory and exploitative experience. Not all sequential experiences contribute equally to the agent’s optimization, and the same experience holds different importance at different learning stages. We propose the Extractor for Adaptive Tradeoff Between Exploration and Exploitation (EATBEE), a task-oriented tool that tightly couples with the agent’s current knowledge and enables adaptive knowledge acquisition. We compare the originally sampled data with the task-driven data distribution to clearly illustrate their deviation. Then, we show how EATBEE identifies and extracts beneficial data for the agent. The monotonic improvement policy is theoretically validated under the assumption that the experience trajectory keeps a high degree of trajectory similarity after extraction. EATBEE serves as an independent module that can be seamlessly integrated with most existing RL algorithms. We substantiate the efficacy and practical applicability of the EATBEE method through experiments conducted in both discrete and continuous environments.