Vulnerability-aware and Curiosity-driven Adversarial Reinforcement Learning Policy for Safety-Critical Scenario Generation

Autonomous vehicles (AVs) face significant threats to their safe operation in complex traffic environments. Adver-sarial policy for scenario generation has been established as a robust paradigm for enhancing AV resilience against adver-sarial perturbations through proactive exposure to synthetically engineered safety-critical scenarios. Training an attacker within an adversarial policy, allowing the target AV to expose vulnerabilities through interaction with this attacker. However, adversarial policies in existing methodologies often get stuck in a loop of over-exploiting established vulnerabilities, resulting in poor exploration for AVs. To overcome the limitations, we introduce a pioneering framework termed the vulnerability-aware and curiosity-driven adversarial reinforcement learning policy. Specifically, during the traffic vehicle attacker training phase, a surrogate network is employed to fit the value function of the AV victim, providing dense information about the victim's inherent vulnerabilities. Subsequently, random network distillation is used to characterize the novelty of the scenario, constructing an intrinsic reward to guide the attacker in exploring unexplored territories. Experimental results demonstrated that the adversarial policy embedded within the attacker exhibited robustness in convergence and significantly enhanced the activation of policy exposure in learning-based AVs, outperforming both other adversarial modalities and alternative reinforcement learning approaches, with a notable reduction in crash rates. The code is available at https://github.com/caixxuan/VCAT.