Self-Organizing Maps-Assisted Variational Autoencoder for Unsupervised Network Anomaly Detection

In network intrusion detection systems (NIDS), conventional supervised learning approaches remain constrained by their reliance on labor-intensive labeled datasets, especially in evolving network ecosystems. Although unsupervised learning offers a viable alternative, current methodologies frequently face challenges in managing high-dimensional feature spaces and achieving optimal detection performance. To overcome these limitations, this study proposes a self-organizing maps-assisted variational autoencoder (SOVAE) framework. The SOVAE architecture employs feature correlation graphs combined with the Louvain community detection algorithm to conduct feature selection. The processed data—characterized by reduced dimensionality and clustered structure—is subsequently projected through self-organizing maps to generate cluster-based labels. These labels are further incorporated into the symmetric encoding-decoding reconstruction process of the VAE to enhance data reconstruction quality. Anomaly detection is implemented through quantitative assessment of reconstruction discrepancies and SOM deviations. Experimental results show that SOVAE achieves F1 scores of 0.983 (±0.005) on UNSW-NB15 and 0.875 (±0.008) on CICIDS2017, outperforming mainstream unsupervised baselines.