Research on big data network intrusion detection based on multiple machine learning models: sampling techniques and performance optimization

In the big data era, network security faces severe challenges, and accurately identifying network attacks and benign instances is of great significance. This study is based on the Intrusion Detection Assessment Dataset and uses three machine learning models, namely Random Forest, XGBoost, and LightGBM, to conduct research on network intrusion detection. During the research process, data preprocessing such as data cleaning, label mapping, and class imbalance handling was carried out first. SMOTE oversampling and undersampling techniques were used to optimize the dataset distribution. The performance of the models was evaluated by comparing and analyzing indicators such as model training time, prediction time, accuracy, precision, recall, F1 - score, as well as confusion matrices, feature importance, and learning curves. The results show that undersampling can shorten the training time, and SMOTE oversampling helps to improve the identification ability of minority classes. Random Forest has a strong ability to handle complex relationships, XGBoost has a stable comprehensive performance, and LightGBM shows a significant improvement in efficiency after parameter adjustment, but the precision of minority classes needs to be optimized. This study provides an effective method for network intrusion detection and also lays a foundation for further research in related fields.