Distributed log anomaly detection method based on improved HDBSCAN

As a critical information source in distributed systems and large-scale IoT infrastructures, log data serves as the foundation for fault diagnosis and security auditing. However, the explosive growth of unstructured and dynamically changing log data poses significant challenges for real-time big data analytics and system scalability. To overcome these performance bottlenecks, this paper proposes and optimizes an end-to-end log anomaly detection method tailored for large-scale log data processing. Specifically, to address template inconsistency across distributed environments, we propose a distributed Drain parsing algorithm featuring dynamic parse tree expansion and global incremental synchronization. Furthermore, we optimize the HDBSCAN algorithm by integrating the RTree spatial index and Boruvka minimum spanning tree (MST). This algorithmic breakthrough effectively eliminates computational bottlenecks in real-time processing and significantly enhances the capacity to capture non-spherical clusters. By proposing a fusion of cross-partition clustering and the GLOSH scoring mechanism, our method strongly supports intelligent analytics in distributed infrastructures. We validate the model on the Spark framework using HDFS, BGL, and OpenStack datasets. Experimental results show that our proposed method has good performance in terms of scalability, resolution efficiency and detection performance, making it highly relevant to modern Internet of Things and large-scale systems.