Audiovisual Speech Improves Speech Recognition and Reduces Mental Effort and Fatigue for Monolinguals and Multilinguals

In adverse listening conditions, speech recognition scores are generally higher when listeners can also see the talker’s mouth. Because bilinguals tend to be more detrimentally affected than monolinguals in audio-only noisy conditions, they may stand to benefit more than monolinguals from a second modality. At the same time, the ability to leverage visual cues may be limited by the amount of exposure a listener has to the target language, which varies among bilinguals. In this study, we aimed to understand whether the benefit of audiovisual speech cues is modulated by listeners’ language experiences. To that end, we measured speech recognition accuracy in noise in both audiovisual and audio-only modalities among monolingual and bilingual listeners. We also collected subjective effort and subjective mental fatigue ratings from participants throughout the experiment to evaluate the relationship between audiovisual speech perception and listening effort for different listeners. Our results show that more active use of another language results in less accurate speech recognition in noise for English, but does not modulate the benefit of the audiovisual speech signal.