Building a Scalable ETL Pipeline with Apache Spark, Airflow, and Snowflake

Extract, Transform, and Load (ETL) pipelines are critical in modern data engineering, enabling efficient data integration and analytics. This paper presents a scalable ETL pipeline leveraging Apache Spark for distributed data processing, Apache Airflow for workflow orchestration, and Snowflake as a cloud-based data warehouse. The proposed architecture ensures fault tolerance, cost efficiency, and high scalability, making it suitable for handling large-scale enterprise data workloads.

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2025-03-21
DOI
https://doi.org/10.5281/zenodo.15125062
Citations
2
Primary Topic
Cloud Computing and Resource Management
Type
article
Field-Weighted Citation Impact
6.49
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Building a Scalable ETL Pipeline with Apache Spark, Airflow, and Snowflake

2 citations
Zenodo (CERN European Organization for Nuclear Research)
Cloud Computing and Resource Management
6.49
article

Building a Scalable ETL Pipeline with Apache Spark, Airflow, and Snowflake

article en
2 citations

Abstract

Extract, Transform, and Load (ETL) pipelines are critical in modern data engineering, enabling efficient data integration and analytics. This paper presents a scalable ETL pipeline leveraging Apache Spark for distributed data processing, Apache Airflow for workflow orchestration, and Snowflake as a cloud-based data warehouse. The proposed architecture ensures fault tolerance, cost efficiency, and high scalability, making it suitable for handling large-scale enterprise data workloads.

Zenodo (CERN European Organization for Nuclear Research)
Industry, innovation and infrastructure
Openalex Percentile: Top 3%
Cloud Computing and Resource Management
6.49
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Building a Scalable ETL Pipeline with Apache Spark, Airflow, and Snowflake · Zenodo (CERN European Organization for Nuclear Research) (2025) | TGRS Research Map | TGRS