Soft Update for Enhanced DQN Based on Mountain Car Problem

The Mountain Car problem in a classical reinforcement learning environment brings great challenges due to its sparse reward and continuous state space. Traditional algorithms such as Q-Learning rely on state discretization, which may introduce problems such as dimensional curse and slow convergence. Deep Q-Network (DQN) addresses these limitations through functional approximation, but instability during training remains a key obstacle. This study systematically evaluates Q-Learning and DQN in the Mountain Car environment, focusing on convergence speed, training stability, and total rewards. To enhance DQN performance, a soft update mechanism based on Polyak averaging is proposed, which smooths target network synchronization, and a novel hybrid reward function. This reward function integrates a sparse success bonus, distance-based penalties, and penalties for oscillatory behaviors near the goal. Experimental results demonstrate that DQN outperforms Q-Learning in terms of convergence speed and stability, highlighting the benefits of deep learning techniques in reinforcement learning. This work provides empirical guidelines for algorithm selection in sparsereward environments and contributes to advancing RL applications in the real world.