The Provenance Gap: What Transport-Layer Capture Sees, What It Cannot, and a Provenance-Native Protocol via the Transport Layer

Data lineage tools learn about data movement from integrations, so code without one, such as anad-hoc script or a notebook, moves data unseen. We study how much of this gap better capture canclose and how much lies beyond the reach of any observer of the wire. We built DCP (DataConnectivity Protocol), which captures lineage inside the PostgreSQL and Kafka client libraries ofunmodified Python programs. On programs with no lineage code, the OpenLineage client emitted 0events while DCP matched every answer key. At scale, DCP's edge-level events reached aprovenance precision of 0.895; the same lineage expressed in OpenLineage's core run model, whichrecords only a run's inputs and outputs, reached 0.480, at equal recall. On a stress set, DCP's recallfell to 0.880, and every miss had one of two causes: a table that does not record who wrote its rows,or a choice made inside the program. We classify the gap into four classes and derive a ceiling forany interceptor that only observes. We then propose the Provenance-Native Protocol, a framingprotocol over TCP in which a data-moving frame without provenance is invalid, reads return thewrites behind their rows, and writes must cite their inputs. A ledger gateway lets it run in front ofstores that keep a per-row writer id. A TLA+ model checks its guarantees and reproduces DCP'sfailure modes. We project the protocol's benefits from these measurements; its costs remain to bemeasured.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24348184
Primary Topic
Scientific Computing and Data Management
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Provenance Gap: What Transport-Layer Capture Sees, What It Cannot, and a Provenance-Native Protocol via the Transport Layer

Dev Bhirin Desai
Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
preprint

The Provenance Gap: What Transport-Layer Capture Sees, What It Cannot, and a Provenance-Native Protocol via the Transport Layer

Dev Bhirin Desai
preprint en

Abstract

Data lineage tools learn about data movement from integrations, so code without one, such as anad-hoc script or a notebook, moves data unseen. We study how much of this gap better capture canclose and how much lies beyond the reach of any observer of the wire. We built DCP (DataConnectivity Protocol), which captures lineage inside the PostgreSQL and Kafka client libraries ofunmodified Python programs. On programs with no lineage code, the OpenLineage client emitted 0events while DCP matched every answer key. At scale, DCP's edge-level events reached aprovenance precision of 0.895; the same lineage expressed in OpenLineage's core run model, whichrecords only a run's inputs and outputs, reached 0.480, at equal recall. On a stress set, DCP's recallfell to 0.880, and every miss had one of two causes: a table that does not record who wrote its rows,or a choice made inside the program. We classify the gap into four classes and derive a ceiling forany interceptor that only observes. We then propose the Provenance-Native Protocol, a framingprotocol over TCP in which a data-moving frame without provenance is invalid, reads return thewrites behind their rows, and writes must cite their inputs. A ledger gateway lets it run in front ofstores that keep a per-row writer id. A TLA+ model checks its guarantees and reproduces DCP'sfailure modes. We project the protocol's benefits from these measurements; its costs remain to bemeasured.

Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon