Pearson Correlation for Efficient Network Anomaly Detection with Quantization on the UNSW-NB15 Dataset

Advances in intrusion detection systems (IDS) leverage deep learning to comprehend complex patterns in data and improve network security performance. However, datasets such as UNSW-NB15 have abundant features, whereas existing techniques ignore the relationships between these features and fail to obtain a simpler understanding. Our research aim is to combine deep learning and Pearson correlation coefficient (PCC) anomaly detection for IDS with the UNSW-NB15 dataset. In the pre-processing stage, we use several methods, such as label encoder, zero imputation, and Winsorization. We perform rolling PCC on the dataset and preserve new features with a certain threshold for the detection step, in which we use deep learning. We include quantization for model compression to increase the efficiency of our model, then model size and accuracy as comparison metrics. The test results show that the average PCC of the 10 best features resulting from rolling PCC dimension reduction is significantly higher than that of the original dataset. Applying deep learning with a rolling PCC dataset shows accuracy, precision, recall, and f1-score of 0.9979, 0.9981, 0.9978, and 0.9979, respectively, with an accuracy loss of 0.2% from the original dataset. Finally, this novel model has a CR of 1.1×. Meanwhile, the size of the best compression model resulting from compaction and quantization is 18.06 kB with CR 13.1×. The application of rolling PCC on IDS shows significant efficiency in the deep learning model.