Fast Capture of Cell-Level Provenance in Numpy
Effective provenance tracking enhances reproducibility, governance, and data quality in array workflows.However, significant challenges arise in capturing this provenance, including: (1) rapidly evolving APIs, (2) diverse operation types, and (3) large-scale datasets.To address these challenges, this paper presents a prototype annotation system designed for arrays, which captures cell-level provenance specifically within the numpy library.With this prototype, we explore straightforward memory optimizations that substantially reduce annotation latency.We envision this provenance capture approach for arrays as part of a broader governance system for tracking for structured data workflows and diverse data science applications.
Authors
- Sanjay Krishnan (ORCID: https://orcid.org/0000-0001-6968-4090)
- Jinjin Zhao (ORCID: https://orcid.org/0000-0001-8536-3510)
Institutions
- University of Chicago (US)
Publication Details
- Published
- 2025-06-22
- DOI
- https://doi.org/10.1145/3736229.3736269
- Primary Topic
- Scientific Computing and Data Management
- Type
- article
- Field-Weighted Citation Impact
- 0.00