A context-aware driver state intervention system via multi-modal fusion and edge-deployed large language models
Abnormal driver states represent a major contributing factor to road traffic accidents. Traditional Driver Monitoring Systems (DMS) predominantly rely on single data modalities and often lack intelligent intervention mechanisms characterized by context awareness. To address these limitations, this study proposes a context-aware driver state intervention system integrated with multi-modal fusion and edge-deployed Large Language Models (LLMs). The system is designed for the real-time monitoring of potential risks—including fatigue, distraction, and physiological anomalies—while providing human-like intelligent interventions. By synchronously extracting visual, inertial, and physiological (heart rate) signals, the framework employs Kalman filtering and Long Short-Term Memory (LSTM) networks to achieve deep fusion of temporal features and robust state evolution analysis. Notably, we leveraged ONNX/TensorRT acceleration technologies to efficiently deploy a lightweight Qwen LLM on an edge device (NVIDIA Jetson Nano). This model performs context parsing and zero-shot inference on multi-source real-time data to dynamically generate targeted, personalized voice alerts. Experimental results demonstrate that the system achieves an integrated detection accuracy exceeding 93% for various abnormal states in complex scenarios, with a perception/detection latency stabilized between 180 and 250 ms (enabling rapid emergency-response latency under 250 ms for critical states), while the complete LLM-generated intervention latency is managed asynchronously at approximately 1.8–2.5 s. This decoupled design significantly outperforms conventional unimodal and static-threshold approaches. This research not only broadens the scope and depth of driver state monitoring but also demonstrates the technical feasibility of transitioning from “passive mechanical alarms” to “proactive intelligent intervention,” offering a promising architectural foundation for the evolution of Advanced Driver Assistance Systems (ADAS) and intelligent transportation safety.
Authors
- Sheng Liu (ORCID: https://orcid.org/0000-0003-2731-9591)
- Licheng Zhuang (ORCID: https://orcid.org/0009-0006-3752-7975)
- Lili Pei
- Jun Hu
- Li Shi
Institutions
- Huaibei Normal University (CN)
Publication Details
- Journal
- Discover Computing
- Published
- 2026-11-21
- DOI
- https://doi.org/10.1007/s10791-026-10878-4
- Primary Topic
- Human-Automation Interaction and Safety
- Type
- article
- Field-Weighted Citation Impact
- 0.00