ProgressiVis: A Language and Environment for Progressive Data Analysis and Visualization

We introduce ProgressiVis, a language and graphical user interface designed to scale interactive visualization and visual analytics by relying on the Progressive Data Analysis (PDA) programming paradigm. Existing systems cannot remain interactive for open-ended exploration with large datasets and complex analyses because their processing time increases with size and complexity; at some point, time exceeds human latency limits (≈10 seconds). PDA systems guarantee that their latency remains within specified limits. They split long computations into smaller execution chunks so that processing a large dataset or applying complex computations still takes time, but they produce intermediary results continuously, improving quality and allowing analysts to monitor the computation, interact with it, and make decisions, often before the whole computation is complete. Implementing PDA systems is not currently supported by any programming environment, forcing programmers to reimplement the data analysis and visualization components from scratch, a long and difficult task. ProgressiVis is a language and system designed to run its programs progressively, including data management, data analysis, and visualizations. It introduces novel programming constructs and provides new services, such as progressive quality assessment, as well as JupyterLab notebook extensions to better support progressive visualization and program execution. This article describes the language and its capabilities for controlling latency. It shows examples of ProgressiVis programs that perform visualization and analysis at scale and explains how it addresses the technical and scientific issues raised by PDA, enabling data exploration to reach new limits.