Effects of voice and gesture interaction with in-vehicle robots on driving performance, driving workload, and user experience: evidence from multimodal physiological measures across driving environments
This study investigates the effects of voice and gesture interaction with in-vehicle robots on driving performance, workload, and user experience using multimodal physiological measures. A 2 (driving environment: simple vs. complex) × 3 (interaction modality: none, voice, gesture) within-subject simulated driving experiment was conducted. Eye-tracking, functional near-infrared spectroscopy (fNIRS), and electrocardiography (ECG) were used to record physiological responses. The results indicate that complex driving environments significantly increase cognitive and physiological demands, as reflected in higher workload, heart rate and pupil diameter, along with reduced blink frequency. The no-interaction condition was associated with lower driving workload and better driving performance. In complex environments, interaction tasks also increased ΔHbO in the dorsolateral prefrontal cortex (DLPFC). Although subjective evaluations favoured voice over gesture interaction, no significant differences were observed in physiological measures, whereas gesture interaction resulted in greater gaze diversion. These findings highlight the value of multimodal physiological measures for evaluating human-vehicle interaction.
Authors
- Zenggen Ren (ORCID: https://orcid.org/0000-0002-3881-0581)
- Fu Guo (ORCID: https://orcid.org/0000-0001-6209-1874)
- Mingming Li (ORCID: https://orcid.org/0000-0002-4702-6350)
- Liting Li (ORCID: https://orcid.org/0000-0002-7478-7534)
Institutions
- Northeastern University (US)
- University of Technology (RU)
- Taiyuan University of Technology (CN)
- Northeastern University (CN)
Publication Details
- Journal
- Ergonomics
- Published
- 2026-11-13
- DOI
- https://doi.org/10.1080/00140139.2026.2746698
- Primary Topic
- Human-Automation Interaction and Safety
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- National Natural Science Foundation of China