Closing the Gaps on Inscrutability: Tackling Challenges with Knowledge Integration during AI development

The development of complex artificial intelligence (AI) systems presents a compelling knowledge integration challenge to organisations. As the organisations strive to integrate complex domain knowledge into algorithmic models, they also have to arm domain experts with the technical understanding of how such models work so they can be used responsibly. The inscrutability of AI technology – stemming from challenges related to both the technical explainability of the models as well as their social interpretability – makes knowledge integration particularly challenging by creating and deepening knowledge gaps between the AI model, its human users and domain reality. To increase understanding of how such knowledge gaps can be addressed in AI development, this study reports on three qualitative case studies on AI projects faced where inscrutability needed to be managed. Building on the gap model (Kayande et al., 2009), we identify three sociotechnical mechanisms for addressing knowledge gaps related to AI inscrutability and thus facilitating organisational learning. Our work provides contributions to both theory and practice.