An end-to-end framework for data lineage analysis covering link pattern recognition, fault diagnosis, and early warning

With the increasing complexity of data platforms, achieving real-time prediction and tracing of data link failures has become a critical issue that needs to be addressed. We proposes an End-to-End Full-Link intelligent analysis framework (EEFL) based on data lineage. This framework combines graph structures with deep learning algorithms to achieve link pattern recognition and fault warning. First, a dynamic data lineage graph model is constructed and topological features are extracted using a graph neural network (GNN). Through temporal edge weight optimization and semi-supervised clustering, typical link patterns are automatically classified. Second, a hybrid fault diagnosis model is designed, using a temporal convolutional network (TCN) to capture long-term dependencies between link metrics and combining it with a GNN to analyze topological mutations. This model accurately classifies various fault types, including data outages, latency anomalies, and data contamination. Finally, a dynamic threshold warning mechanism is introduced, combining Bayesian optimization and online learning to adaptively adjust alarm triggering conditions and effectively reduce false alarm rates. We verifies the generalization ability of the model using actual enterprise data and simulation data. Experimental results show that EEFL can achieve an average Acc of 92.73% across two datasets, which is significantly better than traditional methods and provides intelligent decision for data governance.