An approach for identification of driver drowsiness, emotion state, and behavior patterns based on vision transformers, CNN and multi-modal sensor fusion using RAG-based interpretability

Driver safety is a major issue in modern transportation systems with driver fatigue, distraction, affective state and aggressive driving being significant factors in today’s road traffic accidents. This research proposes a comprehensive multi-modal system using video-based analysis, facial expressions, fatigue analysis and sensor driven driving style prediction to provide real-time driver monitoring. Video stream is processed to take full frame images for behaviour recognition and cropped facial regions are generated for drowsiness and emotion recognition, whilst eye aspect ratio (EAR) and mouth aspect ratio (MAR) metrics are calculated in order to quantify fatigue, and head pose estimation; measuring sustained pitch deviation beyond 0.35 radians and yaw deviation beyond 0.50 radians which is applied to detect drowsiness related nodding and driver distraction. Vision Transformer (ViT—B/16) models with advanced optimisation techniques, scheduler strategy, loss functions and data augmentation protocols (Mixup, CutMix, RandAugment) yield high accuracy in the detection of driver behaviour, drowsiness and emotion. However, robustness evaluation reveals that CNN (convolutional neural networks) architectures outperform or match ViT-B/16 models in certain modules. The final model for each module is therefore selected on the basis of generalization performance. Driving style is evaluated with the help of the sensor data processed through classical machine learning models and temporal modelling through sliding window Long Short-Term Memory (LSTMs) and achieves robust performance even in the sequential prediction tasks. All the outputs, such as EAR, MAR, blink count, head turning, yawn counts, emotion, driver behaviour, and sensor-based classification are integrated in a proposed Retrieval-Augmented Generation (RAG)-based interpretability framework, built upon a domain-specific knowledge base with severity-stratified thresholds, a deterministic penalty-based risk scoring function, and a structured prompt architecture that grounds every recommendation in retrievable, auditable domain knowledge. The RAG system produces a comprehensive driver safety rating (Safe/Needs Attention/Risky), a score out of 100 and reasoning behind the decision, as well as three recommendations for action. Model interpretability and robustness are focused upon using Grad-CAM visualizations. The proposed multimodal framework demonstrates the feasibility of combining deep learning, classical machine learning and explainable AI techniques for real-time, interpretable and reliable driver monitoring, laying the foundation for intelligent transportation systems and advanced driver assistance systems (ADAS).