An Intelligent Real-Time Driver Activity Recognition System Using Spatio-Temporal Features

With the rapid increase in the number of drivers, traffic accidents due to driver distraction is a major threat around the world. In this paper, we present a novel long-term recurrent convolutional network (LRCN) model for real-time driver activity recognition during both day- and nighttime conditions. Unlike existing works that use static input images and rely on major pre-processing measures, we employ a TimeDistributed convolutional neural network (TimeDis-CNN) layer to process a sequential input to learn the spatial and temporal information of the driver activity without requiring any major pre-processing effort. A pre-trained (CNN) layer is applied for robust initialization and extraction of the primary spatial features of the sequential image inputs. Then, a long short-term memory (LSTM) network is employed to recognize and synthesize the dynamical long-range temporal information of the driver’s activity. The proposed system is capable of detecting nine types of driver activities: driving, drinking, texting, smoking, talking, controlling, looking outside, head nodding, and fainting. For evaluation, we utilized a real vehicle environment and collected data from 35 participants, where 14 of the drivers were in real driving scenarios and the remaining in non-driving conditions. The proposed model achieved accuracies of 88.7% and 92.4% for the daytime and nighttime datasets, respectively. Moreover, the binary classifier’s accuracy in determining whether the driver is non-distracted or in a distracted state was 93.9% and 99.2% for the daytime and nighttime datasets, respectively. In addition, we deployed the proposed model on a Jetson Xavier embedded board and verified its effectiveness by conducting real-time predictions.