It Takes Two to Tango – A Comparative Analysis of Human and AI Decision-Making through Eye Tracking and Explainable AI

Understanding the differences and similarities in human and AI decision-making can help illuminate their individual strengths, enabling more effective human-AI collaboration. This study aims to advance this understanding via a comparative analysis of human and AI decision-making. Image classification has proven to be an ideal platform for such analysis, but previous studies have been limited to routine task settings. In contrast, we conduct our comparative analysis in a non-routine task setting, where humans and AI classify Google Street View images. We perform an eye tracking experiment to unveil human decision-making and utilize an explainable AI method to unveil AI decision-making. Our comparative analysis is based on the earth mover’s distance. The study reveals two key findings. First, we find that inter-human similarity is significantly higher than human-AI similarity. Second, human and AI decision-making is the most dissimilar when both the human and the AI make a correct decision.