An eye-tracking study of the interaction of text and facial cues in speech comprehension

Effective speech comprehension in noisy environments depends on integrating auditory and visual cues. This study examined how signal-to-noise ratio (SNR), text accuracy, and facial cues influence fixation behaviors during audiovisual speech comprehension. Twenty-four participants completed trials combining three SNR levels (−4, −5, and −7 dB), with three text accuracy conditions (accurate, two words missing, and three words missing), and two video conditions (facial cues present or absent). The result was the log-transformed fixation duration ratio of the text dwell time to face dwell time. The result was submitted to a linear mixed-effects model which included fixed effects for signal-to-noise ratio, text accuracy, and video display, and random effects for participants and sentences. Text accuracy (χ² = 225.35, p < 0.001) and video presence (χ² = 397.37, p < 0.001) significantly influenced fixation behaviors. Poor text accuracy reduced text dwell time, particularly with three missing words (Estimate = −3.753, p < 0.001). Video presence decreased text fixation, favoring facial cues (Estimate = −4.559, p < 0.001). Under a poor signal-to-noise ratio, low text accuracy amplified reliance on facial cues (χ² = 14.50, p = 0.0007). Therefore, poor text accuracy diminishes its utility, prompting compensatory reliance on facial cues for speech comprehension.