AI-Assisted Triage of Flaky Test Failures from System Logs: A Practical Pipeline for CI at Scale

This study presents a practical and reproducible pipeline for managing flaky test failures directly from CI logs. We parse raw log streams online using Drain/Drain3 to create stable templates, aggregate them within per-failure windows, and vectorize the data using TF-IDF over template n-grams and basic statistics. Next, we apply HDBSCAN to group recurrent failure families into human-readable cluster cards, which include representative examples, dominant templates, and default routing rules. For systems with a strong sequential structure, we can optionally train an LSTM model in the style of DeepLog on template sequences to detect off-pattern executions and identify likely next events. The pipeline is designed for low-label settings, prioritizes explainability, and integrates governance through versioned rules that control actions like quarantine, environment health probes, and owner assignment. We outline a reproducible evaluation plan for teams to use in production contexts, focusing on clustering coverage and purity, the CI dashboard's signal-to-noise ratio (SNR), reductions in median time-to-recover/repair (MTTR), and the rate of duplicate investigations. We provide illustrative (neutral) metrics to demonstrate how to report improvements without revealing proprietary data and discuss a negative case that documents a rule which initially reduced reruns but increased false quarantines; this was later corrected by adding a confirmation step. Overall, by combining robust online parsing, density-based clustering, and optional sequence modeling, we transform noisy failures into explainable, routable families that stabilize delivery at scale while remaining compatible with standard CI/CD tooling.