An Explainable XGBoost-Based Framework for IoT Attack Detection with Unseen Attack Family Evaluation

The rapid growth of the Internet of Things (IoT) has introduced significant cybersecurity challenges due to the heterogeneity, scale, and limited protection capability of connected devices. Although machine learning has been widely adopted for IoT intrusion detection, many existing studies still rely primarily on closed-world evaluation settings, unequal baseline comparison budgets, fixed decision thresholds, and limited integration of explainability into model assessment. To address these issues, this paper proposes an explainable XGBoost-based framework for IoT attack detection with unseen attack family evaluation using the large-scale CICIoT2023 dataset. In the proposed framework, IoT traffic is formulated as a binary classification task that distinguishes benign from malicious flows. The study integrates two complementary evaluation protocols: (1) closed-world stratified 10-fold cross-validation for in-distribution performance assessment and (2) unseen attack family evaluation, in which one malicious family is excluded from training and used only for testing under a zero-day-like but single-dataset condition. A fair-budget experimental design is adopted to compare seven representative models under the same training budget, including default XGBoost, optimized XGBoost, Random Forest, LightGBM, CatBoost, Logistic Regression, and a simple multilayer perceptron. To improve reproducibility and operational validity, the revised framework further reports the sampling strategy, split-overlap audit, XGBoost hyperparameter search protocol, repeated unseen-family evaluation, validation-based threshold calibration under fixed-FAR constraints, cost-sensitive threshold analysis, and XGBoost-native SHapley Additive exPlanations (SHAP) compatible feature contribution analysis. The closed-world results show that tree-based ensemble methods clearly outperform the linear and shallow neural baselines. Random Forest achieves the highest closed-world macro-F1 of 0.9713, followed by LightGBM with 0.9602 and optimized XGBoost with 0.9566. In the fair-budget unseen-family setting under the default threshold, Random Forest again obtains the highest mean macro-F1 of 0.8433 and the lowest false negative rate (FNR) of 0.0712, but it also produces a substantially higher false alarm rate (FAR = 0.0536). By contrast, optimized XGBoost provides a lower-FAR default operating point, achieving a mean macro-F1 of 0.8194, Matthews correlation coefficient (MCC) of 0.7067, FAR of 0.0086, and FNR of 0.2996. Repeated unseen-family experiments over five random seeds confirm the same trade-off: Random Forest provides stronger recall-oriented detection, whereas optimized XGBoost provides a lower-FAR default operating point. After validation-based threshold calibration at an approximate FAR target of 0.01, Random Forest achieves the strongest calibrated recall-oriented performance, with macro-F1 of 0.8754, MCC of 0.7757, FNR of 0.2000, and attack recall of 0.8000. Optimized XGBoost remains competitive at the same FAR target, with macro-F1 of 0.8323, MCC of 0.7193, FNR of 0.2760, and attack recall of 0.7240. The explainability analysis indicates that the optimized XGBoost detector relies mainly on TCP control-flag, temporal, and packet-statistical features, with rst_count, IAT, urg_count, Tot size, Number, Header_Length, and Magnitude among the most influential variables. Local contribution tables for representative true-positive, false-positive, false-negative, and true-negative cases further improve the readability of the explanation results and confirm that native pred_contribs reconstructs the model margin with negligible numerical error. Overall, the results show that the most appropriate model depends on the deployment objective: Random Forest is preferable when minimizing missed attacks under a calibrated FAR constraint is prioritized, whereas optimized XGBoost remains a strong primary model for an explainable low-FAR XGBoost-based framework that emphasizes scalability, operational conservativeness, and native contribution-based interpretation.