The Role of Environment Complexity in Exploration Strategies: A Comparative Analysis of CartPole and Lunar Lander

Exploration strategies play a critical role in the performance of reinforcement learning (RL) agents, especially in environments with varying levels of complexity and reward structures. This paper investigates the differing behaviors of several exploration strategies, including Epsilon-Greedy, Softmax, Upper Confidence Bound (UCB), Thompson Sampling, Boltzmann Exploration, Intrinsic Motivation (Curiosity-driven Exploration), Noisy Networks, Random Network Distillation (RND), Entropy Regularization, and Exploration via Randomized Value Functions (RVF), in two widely used RL environments: CartPole and Lunar Lander. To the best of our knowledge, this is the first work to systematically examine how various exploration strategies interact with the characteristics of these environments using a Deep Q-Network (DQN) as the baseline model. This work contributes to a deeper understanding of the exploration-exploitation trade-off and provides a foundation for selecting suitable exploration methods for simple and complex environments in RL applications. The findings highlight the importance of aligning exploration strategies with the underlying environment dynamics and reward structures, offering insights into the design of RL agents for diverse tasks.