Vision-Language Model Security in Autonomous Driving: A Survey

With the rapid advancement of Vision-Language Models (VLMs), their remarkable capabilities in multimodal perception and decision-making have garnered significant attention in autonomous driving. By integrating VLMs, autonomous driving systems can achieve a deeper understanding of their environment, thereby enhancing safety and efficiency. However, despite their advantages, the deployment of VLMs also introduces potential security vulnerabilities that pose critical challenges to real-world applications. They stem from the complex nature of multimodal processing, making VLMs susceptible to various adversarial manipulations. This paper presents a comprehensive and systematic review of various attack vectors targeting VLMs in autonomous driving, including adversarial attacks, backdoor attacks, jailbreak attacks, prompt injection attacks, zero-shot attacks, and hallucinations. Furthermore, we analyze defensive mechanisms against each attack type and discuss promising future research directions to bolster the robustness and security of VLMs in autonomous driving. By addressing these challenges, we believe that our survey provides insights to the development of safer autonomous driving systems, ensuring their reliability in practical deployments.