Generating Diverse Dangerous Scenarios Considering Real-Time Adversarial Reward for Autonomous Driving Test Using Deep Reinforcement Learning

Autonomous driving must be comprehensively evaluated before deployment, making the generation of dangerous scenarios for the test highly crucial. However, most existing methods only consider terminal state collision reward, resulting in unsolvable scenarios and insufficient non-collision dangers. Additionally, their approaches to increase scenario diversity cause a decline in scenarios quality and requiring for multiple trainings. In this paper, we propose a novel algorithm to generate dangerous scenarios considering real-time adversarial reward based on actor-critic reinforcement learning algorithm. To avoid the defects of collision reward, we take real-time state risk evaluated by time-to-collision into adversarial reward function with consideration of solvability. Meanwhile, in order to increase diversity while keeping scenarios quality and without retraining, we propose only the penalty on the neighborhood of historical actions and single-step differential reward actor-critic algorithm. The results show that our method significantly reduces the proportion of unsolvable scenarios from 32.26% to 7.24% and successfully generates non-collision dangerous scenarios. Furthermore, it achieves similar diversity performance compared to the baseline while reducing the scenario quality loss due to diversity increasing by 64.17%, with only two rounds of training required.