Enhancing passenger-vehicle interaction through multimodal explanation for unexpected behaviors of fully autonomous driving in non-driving-related tasks

• This study investigated the effects of multimodal interfaces and NDRTs in FAVs in various unexpected situations. • Speech explanations increased SA and trust when passengers were not engaged in NDRTs. • Visual explanations enhanced pragmatic quality of UX for NDRT-engaged passengers. • Multimodal XAI interfaces lower UX hedonic quality for NDRT-engaged passengers. • The interface type and NDRTs should be considered in designing an explanation. This study investigates the effects of multimodal explainable artificial intelligence (XAI) interfaces on passenger experience during unexpected behaviors of fully autonomous vehicles (FAVs) when engaging in non-driving-related tasks (NDRTs). An experiment was conducted with 70 participants using a driving simulator. The experiment was designed as a 2 (performing NDRT or not) × 2 (textual explanation or none) × 2 (speech explanation with sound or sound only) between-subject design, focusing on user experience (UX), explanation satisfaction, situation awareness, and trust regarding multimodal XAI interfaces. We included eight unexpected driving situations (i.e., weather type, traffic condition, etc.) that were required as unusual and strange. The findings showed an interaction effect between NDRTs and speech explanations for pragmatic and hedonic qualities of UX. When passengers were in NDRTs, the effectiveness of multimodal XAI interfaces was worse than presenting no explanation at all. When passengers were not in NDRTs, multimodal XAI interfaces improved the overall qualities of UX. Single modality explanations also had a positive effect compared to none, with auditory speech explanations superior to visual ones. These findings have practical implications for supporting effective passenger-vehicle interaction systems that cater to varying user contexts. Specifically, this study demonstrates how well-crafted explanations can enhance trust, situation awareness, and user satisfaction, depending on NDRTs engagement. The results provide actionable insights for developing user-centered design guidelines for XAI interfaces, ensuring a positive UX in FAVs under dynamic and unexpected driving conditions.