Green but Wrong: A Naive Airflow Pipeline Design Failed All 121 Write-Path Fault Trials, a Hardened Design Failed None of 133

An orchestrator can report success while the data it produced is wrong. Apache Airflow executes tasks at least once: a task can be killed after writing half its rows, retried, triggered twice for the same data interval, or re-run after late data arrives, and in each case the DAG run can end green while the sink holds duplicate or stale records. This study measures that gap directly, under a pre-registration frozen before any experiment code existed. On a real local Airflow 3.4.0 deployment (LocalExecutor, SQLite metadata database), five ML pipeline types (ingestion, embedding, evaluation, retraining, monitoring) ran in two arms that differ only in their write and read pattern. The naive arm (append inserts, commit per batch, latest-data reads, wall-clock keys) is the design a straightforward implementation produces. The hardened arm applies three documented design patterns: idempotent sinks with upsert on deterministic keys, one transaction per write task, and fixed-partition reads and writes. Six fault types were injected across 305 logged trials, and after every trial an independent checker re-derived the expected sink content from the corpus specification and counted duplicates, losses, and content mismatches. The naive arm failed every write-path fault trial: 50 of 50 kill-mid-write, 50 of 50 partial-write, 15 of 15 duplicate-trigger, and 6 of 6 late-data trials, 121 of 121 in total, producing 4,060 duplicate records in the kill and partial-write trials alone. Every one of the 100 failed naive kill and partial-write trials still ended in DAG run success: green-but-wrong in 100 of 100 cases. The hardened arm failed zero of its 133 fault trials; pooled over its 50 kill and 50 partial-write trials, the one-sided Clopper-Pearson 95 percent upper bound on its failure rate is 0.0582 per fault. A scheduler restart corrupted neither arm, and a mid-run code deploy corrupted neither arm when the DAGs shipped in a versioned Git bundle: all six Git-bundle runs executed under exactly one code version, while one exploratory run under an unversioned local bundle mixed two versions inside a single run. Hardened recovery from a write fault cost exactly one extra task execution in 100 of 100 trials, with a pooled median recovery time of 6.97 seconds. The failure mode throughout was duplication, not loss: lost records were zero in all 305 trials. All four pre-registered hypotheses were supported. The claims are bounded by the design: SQLite sinks, one machine, a deterministic embedder stand-in, and small trial counts on the platform-level faults, each reported with its exact upper bound.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24328397
Primary Topic
Scientific Computing and Data Management
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Green but Wrong: A Naive Airflow Pipeline Design Failed All 121 Write-Path Fault Trials, a Hardened Design Failed None of 133

Jesun Ahmad Ushno
1 links
Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
preprint

Green but Wrong: A Naive Airflow Pipeline Design Failed All 121 Write-Path Fault Trials, a Hardened Design Failed None of 133

Jesun Ahmad Ushno
preprint en

Abstract

An orchestrator can report success while the data it produced is wrong. Apache Airflow executes tasks at least once: a task can be killed after writing half its rows, retried, triggered twice for the same data interval, or re-run after late data arrives, and in each case the DAG run can end green while the sink holds duplicate or stale records. This study measures that gap directly, under a pre-registration frozen before any experiment code existed. On a real local Airflow 3.4.0 deployment (LocalExecutor, SQLite metadata database), five ML pipeline types (ingestion, embedding, evaluation, retraining, monitoring) ran in two arms that differ only in their write and read pattern. The naive arm (append inserts, commit per batch, latest-data reads, wall-clock keys) is the design a straightforward implementation produces. The hardened arm applies three documented design patterns: idempotent sinks with upsert on deterministic keys, one transaction per write task, and fixed-partition reads and writes. Six fault types were injected across 305 logged trials, and after every trial an independent checker re-derived the expected sink content from the corpus specification and counted duplicates, losses, and content mismatches. The naive arm failed every write-path fault trial: 50 of 50 kill-mid-write, 50 of 50 partial-write, 15 of 15 duplicate-trigger, and 6 of 6 late-data trials, 121 of 121 in total, producing 4,060 duplicate records in the kill and partial-write trials alone. Every one of the 100 failed naive kill and partial-write trials still ended in DAG run success: green-but-wrong in 100 of 100 cases. The hardened arm failed zero of its 133 fault trials; pooled over its 50 kill and 50 partial-write trials, the one-sided Clopper-Pearson 95 percent upper bound on its failure rate is 0.0582 per fault. A scheduler restart corrupted neither arm, and a mid-run code deploy corrupted neither arm when the DAGs shipped in a versioned Git bundle: all six Git-bundle runs executed under exactly one code version, while one exploratory run under an unversioned local bundle mixed two versions inside a single run. Hardened recovery from a write fault cost exactly one extra task execution in 100 of 100 trials, with a pooled median recovery time of 6.97 seconds. The failure mode throughout was duplication, not loss: lost records were zero in all 305 trials. All four pre-registered hypotheses were supported. The claims are bounded by the design: SQLite sinks, one machine, a deterministic embedder stand-in, and small trial counts on the platform-level faults, each reported with its exact upper bound.

Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon