Fast Capture of Cell-Level Provenance in Numpy

Effective provenance tracking enhances reproducibility, governance, and data quality in array workflows.However, significant challenges arise in capturing this provenance, including: (1) rapidly evolving APIs, (2) diverse operation types, and (3) large-scale datasets.To address these challenges, this paper presents a prototype annotation system designed for arrays, which captures cell-level provenance specifically within the numpy library.With this prototype, we explore straightforward memory optimizations that substantially reduce annotation latency.We envision this provenance capture approach for arrays as part of a broader governance system for tracking for structured data workflows and diverse data science applications.

Authors

Institutions

Publication Details

Published
2025-06-22
DOI
https://doi.org/10.1145/3736229.3736269
Primary Topic
Scientific Computing and Data Management
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Fast Capture of Cell-Level Provenance in Numpy

Sanjay Krishnan, Jinjin Zhao
Scientific Computing and Data Management
article

Fast Capture of Cell-Level Provenance in Numpy

Sanjay Krishnan, Jinjin Zhao
article en

Abstract

Effective provenance tracking enhances reproducibility, governance, and data quality in array workflows.However, significant challenges arise in capturing this provenance, including: (1) rapidly evolving APIs, (2) diverse operation types, and (3) large-scale datasets.To address these challenges, this paper presents a prototype annotation system designed for arrays, which captures cell-level provenance specifically within the numpy library.With this prototype, we explore straightforward memory optimizations that substantially reduce annotation latency.We envision this provenance capture approach for arrays as part of a broader governance system for tracking for structured data workflows and diverse data science applications.

University of Chicago (US)
Openalex Percentile: Top 77%
Scientific Computing and Data Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.