Unified Lineage System: Tracking Data Provenance at Scale

As businesses increasingly rely on large-scale data analysis to build products and features, understanding how data flows through heterogeneous storage assets has become a critical challenge. To address these challenges, we present ULS, an end-to-end lineage aggregator designed to track data flows at scale. Our system features a general data model representing data flows between different asset types, a lineage data model that enables navigation across granularities, and techniques to stitch incomplete lineage traces. We also describe methods for processing the lineage graph to customize precision and recall. We implemented ULS at Meta, where its lineage graph contains billions of nodes and edges and supports over one hundred use cases, including routine operations, capacity management and privacy.