Cross-Cloud ML Pipeline Optimization for Big Data and LLM Workloads

Cloud providers now offer specialized accelerator hardware and scalable analytics frameworks, yet the behavior of end-to-end machine learning pipelines across heterogeneous environments remains poorly understood. This paper presents a quantitative multi-cloud study evaluating Spark ETL processing, distributed transformer training, and large language model inference across Amazon Web Services, Microsoft Azure, and Google Cloud Platform. Under controlled dataset scale, model configuration, and GPU/TPU resource equivalence, our cross-provider routing strategy reduced full pipeline operating cost by 38.7% and improved average accelerator utilization by 34.2% without accuracy degradation. We introduce a reproducible orchestration framework, report detailed behavioral differences between A100, H100, and TPU execution paths, and provide evidence-driven guidance for large-scale deployment decisions. Results show that AWS storage locality delivers superior ETL throughput, Azure H100 nodes offer leading model training and inference efficiency, and TPU workloads require additional adaptation overhead. This work establishes the first end-toend cross-cloud empirical benchmark for large model workloads and demonstrates conditions where multiprovider pipeline optimization offers measurable value.