A proposed reinforcement learning approach via discrete control reformulation and multi-step double DQN for adaptive cruise control in electric vehicles

Adaptive cruise control (ACC) plays a critical role in enhancing road safety and energy efficiency in electric vehicles (EVs). Traditional ACC approaches often face challenges in adapting to complex, dynamic driving environments. AI-driven reinforcement learning (RL) has emerged as a promising solution; however, its real-world adoption faces key challenges, including training stability, convergence speed, and robustness in diverse scenarios. This work reformulates the ACC control structure using a simplified action abstraction that unifies throttle and brake into a single scalar variable within a discrete action space. This design enables smooth, human-like driving behavior while allowing the use of simpler and more stable Deep Q-Network (DQN) variants. To address this, we integrate multi-step returns with Double DQN architecture (Double-MS DQN) to accelerate convergence and enhance policy stability. A stochastic scenario generator is also implemented to expose the agent to varied and unpredictable lead-vehicle behaviors during training and evaluation. The results conducted in the CARLA simulator show that the proposed approach achieves significantly faster convergence (up to 73% reduction in training episodes) and reduces headway errors by over 40% compared to standard DQN, Dueling DQN, and Double DQN. The proposed Double-MS DQN demonstrates that adapting the RL control formulation enables high-performance learning with lightweight, scalable algorithms delivering safer and smoother control in practice.