The joinless partition pattern: a spark-native physical execution pattern for multi-dimensional aggregations in apache spark

Abstract Large-scale data aggregation pipelines in Apache Spark typically require multiple join operations to enrich fact tables with dimensional attributes before computing aggregates. Each join induces a costly shuffle operation that redistributes the entire fact table across the cluster. This paper introduces the Joinless Partition Pattern , a novel optimization technique that eliminates intermediate join shuffles by leveraging broadcast variables and partition-local lookups within mapPartitions. We demonstrate that for workloads involving multiple small-to-medium dimension tables (10MB--1GB each), our approach achieves 3--12× speedup and 20--400× reduction in shuffle volume compared to traditional join-based approaches. We provide formal correctness proofs, complexity analysis, and experimental validation on both synthetic benchmarks and three production workloads running on Azure Databricks. Our results show consistent performance improvements across varying data scales and cluster configurations, with particularly strong benefits for aggregation-heavy analytical queries. The pattern has been deployed in production for 6+ months with demonstrated annual savings of \$15,000--\$20,000 in compute costs.

Authors

Institutions

Publication Details

Journal
Distributed and Parallel Databases
Published
2026-11-14
DOI
https://doi.org/10.1007/s10619-026-07464-4
Primary Topic
Advanced Database Systems and Queries
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

The joinless partition pattern: a spark-native physical execution pattern for multi-dimensional aggregations in apache spark

Ruben Alejandro Jaime
1 links
Distributed and Parallel Databases
Advanced Database Systems and Queries
article

The joinless partition pattern: a spark-native physical execution pattern for multi-dimensional aggregations in apache spark

Ruben Alejandro Jaime
article en

Abstract

Abstract Large-scale data aggregation pipelines in Apache Spark typically require multiple join operations to enrich fact tables with dimensional attributes before computing aggregates. Each join induces a costly shuffle operation that redistributes the entire fact table across the cluster. This paper introduces the Joinless Partition Pattern , a novel optimization technique that eliminates intermediate join shuffles by leveraging broadcast variables and partition-local lookups within mapPartitions. We demonstrate that for workloads involving multiple small-to-medium dimension tables (10MB--1GB each), our approach achieves 3--12× speedup and 20--400× reduction in shuffle volume compared to traditional join-based approaches. We provide formal correctness proofs, complexity analysis, and experimental validation on both synthetic benchmarks and three production workloads running on Azure Databricks. Our results show consistent performance improvements across varying data scales and cluster configurations, with particularly strong benefits for aggregation-heavy analytical queries. The pattern has been deployed in production for 6+ months with demonstrated annual savings of \$15,000--\$20,000 in compute costs.

Distributed and Parallel DatabasesVol. 45(1)
Universidad Nacional de La Plata (AR)
Advanced Database Systems and Queries
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon