Effects of voice and gesture interaction with in-vehicle robots on driving performance, driving workload, and user experience: evidence from multimodal physiological measures across driving environments

This study investigates the effects of voice and gesture interaction with in-vehicle robots on driving performance, workload, and user experience using multimodal physiological measures. A 2 (driving environment: simple vs. complex) × 3 (interaction modality: none, voice, gesture) within-subject simulated driving experiment was conducted. Eye-tracking, functional near-infrared spectroscopy (fNIRS), and electrocardiography (ECG) were used to record physiological responses. The results indicate that complex driving environments significantly increase cognitive and physiological demands, as reflected in higher workload, heart rate and pupil diameter, along with reduced blink frequency. The no-interaction condition was associated with lower driving workload and better driving performance. In complex environments, interaction tasks also increased ΔHbO in the dorsolateral prefrontal cortex (DLPFC). Although subjective evaluations favoured voice over gesture interaction, no significant differences were observed in physiological measures, whereas gesture interaction resulted in greater gaze diversion. These findings highlight the value of multimodal physiological measures for evaluating human-vehicle interaction.

Authors

Institutions

Publication Details

Journal
Ergonomics
Published
2026-11-13
DOI
https://doi.org/10.1080/00140139.2026.2746698
Primary Topic
Human-Automation Interaction and Safety
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Effects of voice and gesture interaction with in-vehicle robots on driving performance, driving workload, and user experience: evidence from multimodal physiological measures across driving environments

Zenggen Ren, Fu Guo, Mingming Li, Liting Li
2 links
Ergonomics
Human-Automation Interaction and Safety
article

Effects of voice and gesture interaction with in-vehicle robots on driving performance, driving workload, and user experience: evidence from multimodal physiological measures across driving environments

Zenggen Ren, Fu Guo, Mingming Li, Liting Li
article en

Abstract

This study investigates the effects of voice and gesture interaction with in-vehicle robots on driving performance, workload, and user experience using multimodal physiological measures. A 2 (driving environment: simple vs. complex) × 3 (interaction modality: none, voice, gesture) within-subject simulated driving experiment was conducted. Eye-tracking, functional near-infrared spectroscopy (fNIRS), and electrocardiography (ECG) were used to record physiological responses. The results indicate that complex driving environments significantly increase cognitive and physiological demands, as reflected in higher workload, heart rate and pupil diameter, along with reduced blink frequency. The no-interaction condition was associated with lower driving workload and better driving performance. In complex environments, interaction tasks also increased ΔHbO in the dorsolateral prefrontal cortex (DLPFC). Although subjective evaluations favoured voice over gesture interaction, no significant differences were observed in physiological measures, whereas gesture interaction resulted in greater gaze diversion. These findings highlight the value of multimodal physiological measures for evaluating human-vehicle interaction.

Ergonomics
Northeastern University (US), University of Technology (RU), Taiyuan University of Technology (CN), Northeastern University (CN)
National Natural Science Foundation of China
Human-Automation Interaction and Safety
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon