When You Can't Just Retry: Designing Failover for Production Generative AI

Common LLM-gateway and retry patterns model failover as request-level retry over synchronous completions. Asynchronous generative workloads — including image and video generation and batch inference — break the assumptions underneath that model: jobs can run for minutes, cost exposure can begin once execution starts independently of result delivery, outcomes may arrive via webhooks after the submitting process has exited, and rate limits can reflect account-level concurrency rather than provider failure. This technical report develops a reference architecture in which a durable workflow owns the retry ladder through persisted attempt state, a three-way signal taxonomy (fault, capacity, and indeterminate), conditional-write terminal arbitration, and quota accounting that survives crashes and duplicate delivery. An open-source discrete-event simulator compares three failover strategies across seven synthetic fault scenarios using 30 seeds per cell and bootstrap confidence intervals. Two ablations show that effective failover can mask reliability-policy misconfiguration: success rates remain essentially flat while tail latency, retry amplification, and/or cost absorb the damage. All results are synthetic. No production system, workload, customer data, or internal company metric is modeled or measured. Code and reproducibility: https://github.com/canjianchen/genai-failover-sim