IPA: Inference Pipeline Adaptation to Achieve High Accuracy and Cost-Efficiency

Efficiently optimizing multi-model inference pipelines for fast, accurate, and cost-effective inference is a critical challenge in machine learning production systems. Our work introduces IPA, an innovative online deep learning inference pipeline adaptation system. IPA dynamically configures batch sizes, replication, and model variants to simultaneously optimize accuracy, minimize costs, and meet user-defined latency Service Level Agreements (SLAs). Unlike existing approaches, IPA captures the correlation between configuration changes across multiple pipeline stages, enabling holistic end-to-end optimization of inference pipelines rather than optimizing each stage individually. Key to IPA's innovation is its joint optimization of accuracy and cost in a multi-stage pipeline setting. Although previous work has focused predominantly on cost-aware optimizations, IPA leverages accuracy-aware adaptations, combining autoscaling and model-switching techniques to achieve precise trade-offs between accuracy, cost, and latency. This approach is particularly advantageous in scenarios where heterogeneity in model variants can be exploited to enhance overall pipeline performance dynamically. Through extensive evaluations in a Kubernetes implementation with five real-world inference pipelines (Video Monitoring, Audio Question Answering, Audio Sentiment Analysis, Summarisation Question Answering, and Natural Language Processing), we demonstrate that IPA improves normalized accuracy by up to 35% with a negligible cost increase of less than 5%. The system's effectiveness in achieving granular trade-offs makes it a valuable contribution to the optimization of complex, multi-stage ML inference pipelines. The replication package is available at https://github.com/reconfigurable-ml-pipeline/ipa.