One Engine to Rule Them All: Unifying Cloud Workloads with Argo Workflows

Public cloud providers attract organizations with a promise: every computing service you need, neatly packaged under one roof. Yet those seeking to break free from vendor lock-in and build a sovereign infrastructure face a different reality - navigating a maze of specialized tools: Airflow for data pipelines, SLURM for HPC, Spark for analytics, and a variety of solutions for serverless functions - all using a different interface. We're here to challenge this fragmentation. What if Argo Workflows could be your universal scheduling engine? Through practical examples, we'll demonstrate how the Argo ecosystem - with its container-native workflow engine and robust event system - can consolidate most, if not all, of your compute infrastructure. We'll share our journey of integrating and/or replacing most of our existing scheduling systems with Argo, revealing concrete patterns that preserve workload-specific requirements while drastically simplifying our architecture.