Bi-Clustering Anomaly Detection: A Dual-Stage Clustering Approach Using Bayesian Gaussian Mixture Models (Bi-BGMM)

Anomaly detection is crucial for maintaining the operational integrity and efficiency of complex systems. Conventional methods, however, often fall short in addressing the nuanced and dynamic nature of real-world data. This study introduces the Bi-Clustering Bayesian Gaussian Mixture Model (Bi-BGMM), an innovative approach that advances the state-of-the-art by implementing hierarchical clustering. This method allows for more nuanced detection and scoring of anomalies through a dual-stage clustering process-initially applied to raw data and subsequently to the derived distances to cluster centers. This hierarchical approach enhances the sensitivity of anomaly detection and facilitates the prioritization of anomalies based on their potential impact. Experimental results across several datasets demonstrate that Bi-BGMM significantly outperforms existing methods in terms of accuracy and prioritization capabilities. The approach offers considerable operational benefits, including enhanced network reliability and more effective resource allocation, which are essential for applications demanding high dependability and prompt anomaly resolution. Future research will explore the scalability of Bi-BGMM to other sectors and its integration with real-time data processing systems, aiming to establish a more versatile framework for predictive analytics across diverse industries.