LatentZoom: Seamless Scaling in Generative Latent Space for Visual Exploration of Local Performance in Deep Neural Networks

Deep Neural Network (DNN) evaluation is a classic topic in machine learning. Unlike existing methods that assess a DNN's general performance on pre-collected real-world data, we propose an approach to exploring a DNN's local performance on generated data. The core idea is to train a generative model (GM) on real-world data and visualize the DNN's performance over the GM's latent space (GLS). The GLS contains all generated samples (GSs) of the GM, capturing combinations of features and their variations present in the GM's training set, thereby enhancing the comprehensiveness of testing. Moreover, the plane organizes all GSs into a continuous space with regular variations in features, enabling users to conduct cross-scale exploration to understand the DNN's behavior across different levels of feature granularity. Our research focuses on two challenges in achieving and applying the idea. First, the GLS is unvisualizable due to its high dimensionality and unboundedness. We thus project the GLS onto a plane and visualize the DNN's performance on this plane. The plane serves as a surrogate for the GLS, providing rich visual cues for locating regions containing GSs involving performance-relevant patterns. Second, the plane alone is insufficient for performance exploration. To overcome this, we develop a system based on it. The system comprises two main views that integrate interactions, such as panning and zooming, to enable users to observe any plane region and assess the DNN's performance on the corresponding GSs. We also develop three auxiliary views to facilitate pattern exploration and to diagnose factors that lead to specific performance patterns. Experimental results demonstrate the usability and effectiveness of our approach.