Adaptive ETL: Secure and Cloud Native Framework for Supply Chain Data Management

This study analyzes data management challenges in supply chain operations, focusing on issues with data standardization, security, quality control, and efficiency when integrating information from multiple external sources [1], [2]. The research evaluates cloud computing technology for the solution framework, considering various cloud platforms such as Amazon Web Services (AWS) [18], Azure [17], and Google Cloud Platform (GCP) [19]. Specifically, the study focuses on an AWS-based cloud computing solution and examine three architectures: a pure Apache Airflow implementation, an Airflow-Lambda hybrid, and an Airflow-AWS Glue integration. The analysis explores how each architecture addresses data quality, source integration, operational overhead, cost optimization, and Extraction Transformation and Load Layer (ETL) processing efficiency. While the pure Airflow and Lambda hybrid solutions offer certain advantages, they show limitations in handling complex logistics data processing requirements. The research concludes that the Airflow-AWS Glue hybrid architecture provides the most effective long-term solution, featuring superior ETL capabilities, automated quality checks, and robust error handling. The study provides implementation recommendations and identifies future enhancement opportunities.