Occupant-Aware Decision-Making with Large Vision-Language Model for Autonomous Vehicles

Autonomous driving (AD) has emerged as a transformative technology that holds the potential to free humans from the need for manual driving and provide a safer, more comfortable and efficient driving experience. However, most AD systems make decisions solely based on vehicle dynamics and environmental factors such as road conditions and surrounding vehicles, while the occupant’s mental states, such as subjective feelings and experience, are neglected. As a result, autonomous vehicles (AVs) often fail to meet the occupant’s physical and mental demands, ultimately leading to a compromised driving experience. In this study, we propose an occupant-aware decision-making paradigm (ODP) for AD systems. ODP first perceives the occupant’s physical and physiological states that are closely related to mental states, such as facial expressions and physiological signals, through the occupant monitoring system (OMS). Then, a large vision-language model (VLM) processes the occupant’s physical and physiological states via the chain of thought (CoT) technique to analyze the occupant’s mental states and infer the occupant’s needs. Finally, the VLM makes driving decisions that match the occupant’s demands and preferences. Experimental results show that ODP can make decisions that are significantly better aligned with the occupant’s actual needs than existing methods.