Towards a Better Evaluation of 3D CVML Algorithms: Immersive Debugging of a Localization Model

Abstract As advancements in robotics, autonomous driving, and spatial computing continue to unfold, a growing number of Computer Vision and Machine Learning (CVML) algorithms are incorporating three‐dimensional data into their frameworks. Debugging these 3D CVML models often requires going beyond traditional performance evaluation methods, necessitating a deeper understanding of an algorithm's behavior within its spatio‐temporal context. However, the lack of appropriate visualization tools presents a significant obstacle to effectively exploring 3D data and spatial features in relation to key performance indicators (KPIs). To address this challenge, we explore the application of Immersive Analytics (IA) methodologies to enhance the debugging process of 3D CVML models. Through in‐depth interviews with eight CVML engineers, we identify common tasks and challenges faced during the development of spatial algorithms, and establish a set of design principles for creating tools tailored to spatial model evaluation. Building on these insights, we propose a novel immersive analytics system for debugging an indoor localization algorithm. The system is built using web technologies and integrates WebXR to enable fluid transitions across the reality‐virtuality continuum. We conduct a qualitative study with six CVML engineers using our system on Apple Vision Pro, observing their analytical workflow as they debug an indoor localization sequence. We discuss the advantages of employing immersive analytics in the model evaluation workflow, emphasizing the role of seamlessly integrating 2D and 3D visualizations across varying levels of immersion to facilitate more effective model assessment. Finally, we reflect on the implementation trade‐offs and discuss the generalizability of our findings for future efforts in immersive 3D CVML model debugging.