From Alert Fatigue to Trusted Insights: A Simple Non-ML Anomaly Detection Experience

Distinguishing actionable anomalies from noise in monitoring metrics is a critical operational challenge. In our experience, static thresholds often resulted in overwhelming alert fatigue, while pilot projects utilizing ML-based models proved to be uninterpretable. We implemented an interpretable system based on simple rules co-designed with our support team, focusing on sustained, meaningful deviations from a seasonal baseline. The result was a dramatic shift in operator trust, with feedback indicating that "all alerts are good".