A real-time multimodal framework for driver behavior recognition using transformer and reinforcement learning

Abstract Advanced Driver-Assistance Systems (ADAS) and other in-vehicle technologies often fail to capture critical indicators of unsafe driver behavior, such as fatigue, distraction, or emotional states. Conventional approaches rely on a single data modality (e.g., video, audio, or vehicle sensors), which limits their effectiveness in real-world conditions. Multimodal driver behavior recognition addresses these limitations by integrating complementary data sources to provide a holistic view of the driver’s state. This paper introduces a dependable, real-time multimodal driver behavior recognition system that combines RGB video, acoustic signals, and geometric keypoints to improve transportation safety. The proposed framework employs lightweight transformer models, efficient video token thinning, noise-aware audio processing, and reinforcement learning-based dynamic path selection to balance accuracy and latency. Evaluated on two standard datasets, the system achieves 99.92% accuracy while operating at 41 frames per second with a latency of only 24 milliseconds, demonstrating its suitability for enhancing driver assistance and autonomous driving systems.