Accepting Mortality: Strategies for Ultra-Long Running Stateful Workloads in K8s

"Pods are mortal" is a well-known quote in the official Kubernetes documentation. For ultra-long running stateful workloads that take months to complete, this mortality comes with its own challenges. How do you react to hardware failures? What resource quotas are appropriate? What if the workload has no built-in persistence and does all its work in memory? For such workloads, failures can be fatal, potentially wiping out months of work. This session will show that despite all the obstacles, Kubernetes can still be a reasonable choice for running stateful workloads that take months to complete. Using real-world examples based on production workflows, we will show how we design, configure, run, and operate such workloads using K8s and Argo workflows. We will also show how intelligent checkpointing using CRIU can help us deal with failures and enables us to avoid some problems even before they occur.