Towards Predictive Schema Intelligence: Predicting Downstream SQL-Pipeline Failures from Schema Changes on an Execution-Grounded Synthetic Benchmark

Changes to upstream database schemas can break downstream datapipelines, and common safeguards such as schema-compatibility rulesand data validation either flag changes regardless of whetheranything downstream depends on them or react only after data hasarrived. This paper asks whether the characteristics of a proposedschema change, together with static lineage information, can predictpipeline failure before the pipeline runs. We build a reproducible, fullysynthetic, execution-grounded benchmark: 500 synthetic pipelines,each with 3–8 downstream SQL jobs, and 10000 proposed schemachanges (column drop, rename, add, type change, nullability relax/tighten). Every change is applied and every job executed in DuckDB2.0.0, so failure labels are observed outcomes rather than handwritten rules; 9614 changes were executable and 386 were rejectedby the engine at migration time and excluded. Under pipeline-grouped5-fold cross-validation repeated three times, tree-based models usingchange features plus static-lineage features reach F1 ≈ 0.999 ± 0.001(Random Forest), compared with 0.797 ± 0.009 for an a-priorilineage-aware rule baseline, 0.587 ± 0.012 for a schema-only rule,and 0.680 ± 0.015 for a Random Forest restricted to change features.The advantage over the lineage-aware rule is concentrated in typechanges, where the rule over-flags. Performance degrades as lineageis hidden (Random Forest F1 0.795 when each job is missing fromlineage with probability 0.5), and models trained without a changetype do not reliably generalize to it. The near-perfect scores show thatfailures in this benchmark are close to a deterministic function of1change, lineage and engine semantics; the results are specific to thesynthetic generator and to DuckDB and do not establish performanceon real pipelines Code, data and results: https://github.com/8abhay/predictive-schema-intelligence