A Multi-modal Drowsy Driver Detection with Random Forest using Video-based and Physiological Features

Driver drowsiness is a common occurrence in road accidents, affecting road safety. This study proposes a hybrid multi-modal data fusion approach that combines facial video features with electroencephalography (EEG) and electrocardiogram (ECG) signals to improve the accuracy and robustness of drowsy driver detection. Using the DROZY dataset, our system extracts spatio-temporal features from both video and physiological data, then merges these features to improve accuracy. The resulting dataset was used to train a model utilizing the random forest algorithm (RF) as the classifier and principal component analysis (PCA) with 95% explained variance for dimensionality reduction. The results show that a higher performance was achieved by combining video-based features and physiological information from EEG and ECG; peaking at around 93%-94% in all performance metrics with the multi-modal approach. Moreover, the effect of varying window sizes (5s, 10s, 30s, 60s) was also investigated, wherein the 30s window generally showed the optimal performance results, while only being edged by the 10s window under the multi-modal scheme. To combat the loss of temporal resolution in increasing window sizes, a sliding window technique may be applied in future iterations of multi-modal DDD research, finding a balance between resolution and computational complexity trade-offs.