Canary: Detecting and Localizing Faults in Data Center Networks With Partial Traffic Monitoring

Silent packet drops and packet corruptions, which are caused by faulty network elements and hurt performances of cloud applications, are common in data centers but hard to detect and localize. Existing solutions based on active probes introduce additional probe traffic and are constrained by probe rate, while solutions based on passive traffic monitoring measure the entire network traffic, and are generally unable to pinpoint the locations where packet losses happen. In this paper, we present <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Canary</i>, a system for detecting and localizing network faults with partial traffic monitoring. Canary employs a lightweight and adaptive mechanism to detect packet losses by monitoring a small set of large-sized network flows, and it ensures that on each network path, a sufficient number of packets are monitored by upstream and downstream switches. In addition, Canary encodes information of the path that a packet travels along within its header, and by leveraging path information of the lost packets, Canary is capable to localize network faults with high accuracy. We theoretically prove the effectiveness of our proposed method, and prototype Canary with P4 on commodity hardware programmable switches. Results from extensive experiments driven by real-world traffic show that Canary is lightweight regarding measurement overhead, robust under traffic dynamics, and is accurate in detecting and localizing faulty network links. In particular, comparing with the state-of-the-art solutions, Canary reduces the memory overhead by over 97% under <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$10^{-2}$</tex-math> </inline-formula> link loss rate, and increases the F1-score in localizing the faulty links by over 20% on a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k=8$</tex-math> </inline-formula> fat-tree data center network.