A Study of Response Time Instability of Microservices at High Resource Utilization in the Cloud

Maintaining consistently low response times is a critical performance requirement for mission-critical, web-facing applications (e.g., e-commerce) that typically use microservices architectures. Through extensive benchmarking of a microservices application in a cloud environment, we demonstrate that its response time stability is fragile, with significant variations (ranging from milliseconds to seconds) when the CPU utilization of component servers reaches moderate to high levels (e.g., 60%). Our detailed timeline analysis identifies a leading cause of response time instability at these utilization levels: a ripple effect caused by the long chain of dependencies inherent in microservices applications. In such an architecture, a millibottleneck (a bottleneck lasting sub-seconds) can trigger a queuing effect from a downstream server that propagates to upstream servers, resulting in dropped requests and TCP retransmissions lasting several seconds at the weakest link in the chain. We demonstrate two common factors that contribute to millibottlenecks in microservices runtime environments: interference from collocated containers and bursty workloads, both of which are prevalent in real-world cloud environments.