Anomaly Detection in Imbalanced Network Traffic Using a ResCAE-BiGRU Framework

To address the critical challenge of low detection rates for rare anomaly classes in network traffic, a problem exacerbated by severe data imbalance, this paper proposes a deep learning framework for anomaly detection in imbalanced network traffic. Initially, the framework employs the Isolation Forest (iForest) and SMOTE-Tomek techniques for outlier removal and data balancing, respectively, to enhance data quality. The model first undergoes unsupervised pre-training using a symmetrically designed Residual Convolutional Autoencoder (ResCAE) to learn robust feature representations. Subsequently, the pre-trained encoder is integrated with a Bidirectional Gated Recurrent Unit (BiGRU) to capture temporal dependencies within the traffic features. During the fine-tuning phase, a Sharpness-Aware Minimization (SAM) optimizer is employed to enhance the model’s generalization capability. The experimental results on the public CICIDS2017 and UNSW-NB15 datasets reveal the model’s outstanding performance, achieving an accuracy, precision, recall, and F1-score of 99.33%, 99.53%, 99.33%, and 99.41%, respectively. Comparative analysis against baseline models confirms that the proposed method not only surpasses traditional machine learning algorithms but also holds a significant advantage over contemporary deep learning models. The results validate that this framework effectively resolves the issue of low detection rates for rare anomaly classes caused by data imbalance, offering a powerful and robust solution for building high-performance anomaly detection frameworks.