Modified Variational Autoencoder and Attention Mechanism‐Based Long Short‐Term Memory for Detecting Intrusions in Imbalanced Network Traffic

ABSTRACT The internet and communication industries have grown at a very quick pace, which has caused a massive increase in the volume of data and network size. This surge has given rise to a multitude of new attacks, posing substantial challenges for network security in effectively identifying breaches. To counteract these threats, intrusion detection systems (IDS) have been created, utilizing technology to scrutinize, monitor, and analyze network traffic and ensure the conservation of availability, confidentiality, and integrity. In networks with imbalanced traffic, malicious cyber‐attacks can easily go unnoticed within large volumes of regular data. This proficiency in concealing their presence poses a formidable obstacle for Network IDS in accurately and promptly detecting such threats. Despite extensive research efforts, conventional IDS proposed models are faced with persistent issues of enhancing detection accuracy and lowering false alarm rates, identifying emerging rare and zero‐day intrusion types. Previous research has also emphasized the problem of uneven distribution in network traffic, potentially leading to the misclassification of attacks. As a solution to these problems, this study proposed a multi‐model architecture that leverages attention mechanism‐based long short‐term memory (AM‐LSTM) and class‐wise focal loss‐based variational autoencoder (CWFL‐VAE), which both aimed to detect various forms of attacks, including rare or zero‐day attacks, while reducing false alarm rates and computational complexity. CWFL‐VAE was employed to handle imbalanced network traffic, focusing on minority classes to address the issue of misclassification; AM‐LSTM was used for classification, while the Adam gradient descent technique was employed to optimize the model. The proposed system performance was assessed using two datasets: NSL‐KDD, a benchmark dataset with skewed network traffic distribution, and CSE‐CIC‐IDS2018, featuring network traffic that is approximately 83% benign cases. CSE‐CIC‐IDS2018 was employed in assessing the performance of the model due to its recent release and incorporation of contemporary attack types, while the NSL‐KDD data functioned as a trustworthy benchmark, testing the model's implementation against findings in the literature. The research showed good performance with a low false positive rate of 0.12%, 99.37% accuracy, and 99.23% detection rate for the NSL‐KDD data. Similarly, the technique's detection rate, accuracy, and false positive rate for the CSE‐CIC‐IDS2018 data were 94.2%, 0.22%, and 92.39%, respectively. According to these findings, the recommended model was found to be competitive in terms of precision, rate of detection, and incidence of false positives when evaluated with existing methods.