Semi-Markov Options Enabled DDPG Method for Autonomous Vehicle Overtaking With LiDAR and RADAR Fusion

Autonomous Vehicle (AV) navigation in dynamic environments is highly complex, with overtaking maneuvers adding complexity due to multiple sub-maneuvers and limited environmental perception. Recent advancements in Deep Reinforcement Learning (DRL) and sensors address a few challenges. Yet, the high dimensional sensor data and complex maneuvers slow RL agents in effectively perceiving the environment and performing the overtaking. Furthermore, the dynamic conditions and individual sensor data limit the RL agent's ability to localize and execute the precise sub-maneuvers. Thus, in this work, we propose TDRLO, a Deep Deterministic Policy Gradient (DDPG) based semi-Markov options acquired hierarchical reinforcement learning framework for AVs overtaking with LiDAR-RADAR fusion to tackle this issue. Our approach uses the option policy to control the sub-maneuvers execution and preprocesses sensor data for extracting crucial overtaking checkpoints and low dimensional environmental perception. Temporal Difference (TD) updates in the DDPG algorithm enhance episodic learning and incremental updates, while the option policy reduces the overtaking complexity. Moreover, the preprocessing of raw data and sensor fusion provides a better environmental perception and efficient overtaking in the CARLA simulator. We evaluated our approach using the National Highway Traffic Safety Administration (NHTSA) inspired overtaking pre-crash scenarios in CARLA. The result shows an average 20-30% improvement in average peak reward with stable critic values, alongside 100% completion rate, 10-30% least collision rate, and 20-40% more average speed compared to the baseline sate-of-the-art methods.