SafeDrive: Knowledge- and data-driven risk-sensitive decision-making for autonomous vehicles with Large Language Models

Recent advancements in autonomous vehicles (AVs) leverage Large Language Models (LLMs) to perform well in normal driving scenarios. However, ensuring safety in dynamic, high-risk environments and managing safety-critical long-tail events remains a significant challenge. To address these issues, we propose SafeDrive, a knowledge- and data-driven risk-sensitive decision-making framework, to enhance AV safety and adaptability. The proposed framework introduces a modular system comprising: (1) a Risk Module for comprehensive quantification of multi-factor coupled risks involving driver, vehicle, and road interactions; (2) a Memory Module for storing and retrieving typical scenarios to improve adaptability; (3) a LLM-powered Reasoning Module for context-aware safety decision-making; and (4) a Reflection Module for refining decisions through iterative learning. By integrating knowledge-driven insights with adaptive learning mechanisms, the framework ensures robust decision-making under uncertain conditions. Extensive evaluations on real-world traffic datasets characterized by dynamic and high-risk scenarios, including highways (HighD), intersections (InD), and roundabouts (RounD), validate the framework's ability to enhance decision-making safety (achieving a 100% safety rate), replicate human-like driving behaviors (with decision alignment exceeding 85%), and adapt effectively to unpredictable scenarios. The proposed framework of SafeDrive establishes a novel paradigm for integrating knowledge- and data-driven methods, highlighting significant potential to improve the safety and adaptability of autonomous driving in long-tail or high-risk traffic scenarios. Project page: https://mezzi33.github.io/SafeDrive/.