Human-AI Co-Construction of Interpretable Predictive Models: The Case of Scoring Systems

This study explores the co-construction of probabilistic scoring systems. Using a self-developed web-based tool, called PSLVIS, participants were able to create their own decision-support models through an interactive interface. Seven academic advising experts participated, assessing the probability of student success both with and without the assistance of a Probabilistic Scoring List (PSL). The results indicate that while the co-constructed models slightly improved the experts’ accuracy, they also increased decision time. Experts interacted with PSLVIS and PSL in diverse ways, displaying different levels of algorithmic aversion and appreciation. This study underscores the potential of decisionsupport systems that integrate data-driven algorithms with human expertise, while also revealing the wide range of challenges that need to be addressed for successful co-construction and practical implementation.