A Mixed Methods Approach to Analyzing the Role of AI Teammates in Transition Phases

Recent innovations in AI have allowed AI agents to work as collaborative teammates; however, these human-AI teams still face significant challenges in achieving high levels of effective coordination and collaboration. Focusing on the temporal nature of teaming, collaboration is often evaluated and improved through transition-phase discussions, held among teammates before or after achieving team goals. Using a mixed-methods approach, this study examines how AI teammates participate in transition-phase discussions and share various types of information, with a focus on situation awareness, impact on team cognition, trust, and performance in human-AI teams. Data from 31 teams completing a three-hour simulated uncrewed aerial system task, comprising four distinct rounds and two transition phases, were analyzed. Quantitative results indicated that AI involvement later in the team's life cycle fostered more trust in the AI teammate, as compared across the four rounds, and was associated with higher performance. Perceived team effectiveness also improved following transition phases the AI teammate participated in, irrespective of whether it occurred early or late in the team's life cycle. Qualitative findings revealed that AI involvement benefits transition phases, particularly when it prompts teammates to recall task details and develop shared knowledge. Based on these results, we demonstrate the value of AI teammates engaging in transition-phase discussions for human-AI teams and provide design recommendations for researchers and practitioners to improve the efficacy of HATs by implementing transition phases.