Evaluating Machine Learning Models for Intrusion Detection Systems in IoT Devices: An Experimental Study

This paper compares machine learning systems to detect network attacks in IoT systems based on the UNSW-NB15, Bot-IoT, and TON_IoT datasets. These data sets have real and artificial samples of network traffic marked with various features and attack labels. Three types of attacks are chosen to be evaluated through multi-class: Denial of Service (DoS), Backdoor, and Reconnaissance. The models that are implemented are Support Vector Machine (SVM), tree-based (LightGBM, CatBoost) and TabNet. Preprocessing included missing value, categorical feature encoding, and sampling strategies that covered the issue of class imbalance. In the case of the multi-class problem on UNSW-NB15, TabNet-L obtained a macro-recall of 77.0% ± 0.8 and macro-F1 of 60.0% ± 0.7 compared to SVM (macro-recall: 71.0% ± 1.1). TabNet-L in binary classification (attack vs. benign) had almost perfect attack recall (99.9) but lower precision (51.2) and hence a high false positive. TabNet-L also continued to perform well on IoT-native data (Bot-IoT, TON_IoT, macro-F1: 92.1% and 85.3%, respectively). The findings prove that TabNet is effective, however, it is important to note that there is the significant problem of false positives and the challenge of classifying minority classes, such as the Backdoors (recall: 37.0%). Reduction strategies and work strategies in the future are discussed.