Human Assessment of AI Errors and its Impact on Hybrid Teaming for Decision Making

With increasing opportunities for collaboration between humans and AI, a crucial aspect of effective teaming is humans' ability to develop mental models (MMs) of their automated teammates. MMs are beneficial in discerning when to accept the agents' recommendations by identifying the error boundaries and overcoming AI brittleness. Participants in our study were tasked as disaster relief planners, assisted by a decision aid that recommends resource allocations based on several information attributes. They were asked to identify the source of the agent's errors, for low and high error complexity levels, across which we measured team performance, participants' ability to identify and compensate for errors, and their subjective experience of working with the AI teammate. Results revealed that high error complexity degraded performance and participants' ability to form accurate MMs. While less complex errors caused satisficing among participants, more complex errors led to active compensation. We also found that most participants underrated the agent for high-complexity errors while overrating an imperfect agent for less complex errors. These findings motivate further research toward understanding how MM formation is impacted due to various levels of automation error and its impacts on human-agent collaborative decision-making.