Can I Trust You? Exploring the Impact of Misleading AI Suggestions on User's Trust

Extended Reality (XR) offers immersive and interactive training experiences across fields like policing and medicine. However, tailoring training difficulty to individual needs remains a challenge. AI-driven training assistants could be a promising solution to personalize XR training by proposing adjustments based on user data. Yet, the effectiveness of such suggestions and the user's ability to discern accurate recommendations are not fully understood. This study investigates the effectiveness of suggestions within an AI-driven interface for proposing scenario changes in an XR training environment. We employ a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2\times 2$</tex> experimental design, manipulating both the presence of stress during training (stressful versus neutral scenarios) and the biosignal information provided by the trainees during training (congruent versus incongruent). This work aims to determine whether trainers can reject misleading suggestions made by the system and how this affects their trust in the system. A sample of N=17 medical first responder trainers was recruited from three European countries. Our results show that the majority of the participants did not reject incorrect suggestions and the expected decrease in trust and perceived transparency towards the misleading interface was not observed. These results underscore the complex interplay between user perception, trust, and technological interface design, highlighting avenues for further research and refinement in the development of AI-driven suggestion systems.