Equipping with Human Cognition: Driver Intention Recognition with Multimodal Information Fusion

To address the issue of autonomous driving systems’ inability to timely detect danger and respond in urban environments, an intention recognition system integrating driver cognitive information was developed. By utilizing the drivers’ electroencephalogram (EEG), eye movement and operational data, the system identifies driver intentions in a type of hazardous scenario. Initially, a driver-in-the-loop simulation platform was used for data collection, followed by experiments and data preprocessing to create a multimodal fused dataset through feature-level fusion. Models based on multilayer perceptron (MLP), convolutional neural network (CNN) and transformer were then developed to predict emergency braking and steering evasion intentions. The transformer-based model, with multimodal data fusion, achieved the best performance with an accuracy of 93.02%, significantly surpassing EEG-only (80.80%) and eye movement and operational data-only models (78.46%). This highlights the transformer’s superior ability to capture complex spatiotemporal correlations in multimodal data. Additionally, pre-extracted EEG frequency domain features could improve model performance, though less significantly than changes in model architecture. Embedding this system into autonomous driving systems is expected to enhance their ability to quickly and accurately recognize and respond to dangerous scenarios.