Uncertainty-Aware Zero-Day Botnet Detection for IoT Networks with Real-World Edge Deployment on Resource-Constrained Hardware

Closed-set evaluation is the most common method in Internet of Things (IoT) sensor network botnet attack detection research. In this setting, classifiers are tested only on attack families encountered during training. As a result, they often fail to detect genuinely new (zero-day) malware in real-world environments. This study evaluates a calibrated two-tier detection pipeline combining an XGBoost gradient-boosted tree classifier for known-class prediction with a Monte Carlo (MC) dropout multilayer perceptron (MLP) as an independent uncertainty estimator, under conditions closer to real deployment than most prior evaluations. The uncertainty model is used to flag unfamiliar traffic that may represent zero-day traffic. The classifier is trained and calibrated on the Bot-IoT dataset and evaluated for zero-day generalization on N-BaIoT, which contains the previously unseen Mirai and BASHLITE malware families. During data preparation, a severe train/test duplication problem was identified. Around 97% of a naive split was found to be duplicates and was removed before splitting. The base classifier achieved 99.4% accuracy in known-class detection. However, the confidence scores were negatively correlated with zero-day traffic, producing an area under the receiver operating characteristic curve (AUROC) of 0.32. This indicates the overconfident misclassification of unseen attacks. The proposed uncertainty gate improved zero-day discrimination to an AUROC of 0.66, with catch rates reaching 13.75% for Mirai. An explainable AI analysis using Shapley Additive Explanations (SHAP) linked this gap to packet size statistics. To evaluate practical feasibility, the complete framework was deployed on an embedded Raspberry Pi Compute Module 5 (CM5) edge device. The full uncertainty-gated pipeline achieved mean inference latency of 15.03 ms per sample. Overall, these findings demonstrate that uncertainty-aware gating can improve zero-day robustness over conventional closed-set classification in resource-constrained IoT sensing environments.