Multi‐Modal Anomalous Driving Behavior Detection With Adaptive Masking

ABSTRACT Intelligent Transportation Systems are tasked with enhancing road safety, a crucial challenge given that approximately 1.35 million fatalities occur globally each year, with 15%–27% of these deaths attributed to Anomalous Driving Behaviors (ADBs). Detecting these behaviors in real time is vital for preventing accidents and improving traffic safety. However, the complexity of driving environments, characterized by diverse scenarios, drivers, and vehicle conditions, makes ADB detection a challenging task. This article proposes a novel approach for ADB detection, leveraging the advantages of multimodal data, adaptive masking, and multihead self‐attention mechanisms. The proposed method first employs an adaptive masking technique based on the Softmax function to sparsify input features, effectively reducing the influence of irrelevant information. By focusing on key features, the model becomes more resilient to noise, such as background clutter or irrelevant driver actions, which might otherwise interfere with the detection of abnormal behaviors. To further enhance feature integration across different data modalities (e.g., visual, infrared, and depth data), a multihead self‐attention mechanism is incorporated. This mechanism enables the model to prioritize important information from various sensor inputs, fostering more effective multimodal fusion and better decision‐making for behavior classification. In addition, a supervised contrastive learning strategy is utilized to mitigate memory usage, a common challenge in real‐time systems where computational resources are limited. This approach ensures efficient learning by emphasizing the distinction between normal and abnormal behaviors while minimizing the memory footprint of the model. Extensive experiments on two benchmark datasets, 3MDAD and DAD, demonstrate the proposed method's superior performance in detecting ADBs. The results indicate a significant improvement in detection accuracy and robustness, highlighting the potential of this approach for deployment in real‐world Intelligent Transportation Systems aimed at enhancing road safety. This research provides a promising step forward in the development of more effective and scalable solutions for ADB detection, offering a foundation for future advancements in traffic safety technologies.