No Crossover Inside the Ladder: A Pre-registered Comparison of DuckDB, Spark Local Mode, and Incremental dbt on a TPC-H-Derived Workload, SF0.1 to SF10

"Just use DuckDB" and "you need Spark" are both common advice for batch transformation work, and both are usually given without numbers. This study pre-registered six hypotheses (PREREG.md, committed before any data was generated) and measured where a single-node embedded engine stops dominating a local-mode distributed engine on one fixed workload: six transformations (filter, join, aggregate, window, type conversion, sort and limit) over TPC-H-derived data at scale factors SF0.1, SF1, and SF10, run identically on DuckDB 2.0.0, on Apache Spark 4.0.1 in local mode, and as a dbt project (dbt Core 1.12.6) on the same DuckDB engine, on one 4-core machine with 7.7 GiB of RAM. DuckDB was faster on the total workload at every scale tested: the total time ratio (DuckDB divided by Spark) is 0.125 at SF0.1, 0.162 at SF1, and 0.248 at SF10, and no single transformation at SF10 reaches the pre-registered 1.5 crossover ratio (the best is 0.467), so no crossover is observed inside the ladder tested (H1 supported, H2 refuted). Correctness held everywhere it was tested: output checksums agree across all three arms in 18 of 18 (scale, transformation) cells, with run-to-run determinism on re-runs (H3 supported). Incremental dbt models with unique_key reproduced the full-refresh checksums under four late-data scenarios at every scale (H4 supported), and dbt data-quality gates configured as promotion gates caught 10 of 10 injected faults with zero missed critical faults (H5 supported). The incremental saving against full refresh is 87.1 percent at SF10, but SF1 also exceeds 50 percent (70.5 percent), where the pre-registration predicted small scales would stay below it, so the economics hypothesis is only partially supported (H6). The data is derived from TPC-H; these are not official TPC-H results and no comparability to official TPC-H results is claimed.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24328271
Primary Topic
Advanced Database Systems and Queries
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

No Crossover Inside the Ladder: A Pre-registered Comparison of DuckDB, Spark Local Mode, and Incremental dbt on a TPC-H-Derived Workload, SF0.1 to SF10

Jesun Ahmad Ushno
1 links
Zenodo (CERN European Organization for Nuclear Research)
Advanced Database Systems and Queries
preprint

No Crossover Inside the Ladder: A Pre-registered Comparison of DuckDB, Spark Local Mode, and Incremental dbt on a TPC-H-Derived Workload, SF0.1 to SF10

Jesun Ahmad Ushno
preprint en

Abstract

No abstract available for this paper.

Zenodo (CERN European Organization for Nuclear Research)
Advanced Database Systems and Queries
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon