A Multi-Stage IoT Security Framework with Hybrid Detection, Contextual Topology Analysis, and Policy-Based Response

In this paper, the authors propose a complete multiclass traffic classification model with an unknown-traffic screening method based on anomaly detection using Isolation Forest and a topological awareness method for context-based risk analysis, a dynamic trust scoring evolution, and a policy-based security action method. Topology is not fed into the traffic classifier; it is used as a contextual risk layer. The contribution is not a new classifier: XGBoost is a very popular gradient-boosted tree classifier, and the good results achieved on CICIoT2023 is a benchmark result, not a contribution. The accuracy of XGBoost is 98.99%, the Macro-F1 is 71.51%, the macro-average attack detection rate (ADR) is 70.98%, and the false positive rate (FPR) is 0.31% on CICIoT2023. Classes with high occurrence rates (e.g., DDoS, DoS, Mirai, and Vulnerability Scan) were easy to detect (F1 ≥ 90%), whereas rare classes (e.g., Uploading Attack, XSS, Backdoor Malware, SQL Injection, Browser Hijacking, Command Injection, and Recon-PingSweep, which were all below 10%) were hard to detect. The hybrid unknown-traffic module was evaluated using the leave-one-family-out method: that is, the attack family was selected and removed from the training dataset of the Isolation Forest, and then served as an unknown set for the test. The detection performance is highly family-dependent: 67 to 95 percent UTDR (new categories of behaviorally different attacks) versus only 0.1 to 3.4 percent UTDR (new variants of volumetric flood attack) with a macro-average of 51.12 percent (±43.69) and a micro-average of 2.18 percent. This dataset has a very imbalanced base rate among classes, leading to low precision for all classes (0.48 − 16.57%). The module is therefore useful to detect behaviorally novel traffic, but not necessarily novel flood-type variants. The unknown-traffic module is a high-sensitivity screen with low precision (0.48 − 16.57%); it is not a reliable unknown-threat detector. Topology analysis identified 120 nodes and 342 links (14 high risk and eight critical), and an average risk score of 0.42 was calculated. Over 60,000 samples were trusted and analyzed with respect to policy; 94.25% of observations remained in the High Trust band and 97.16% of these were attack-labeled. The trust model is thus reported as a negative result: the trust bands were unable to support a reliable benign/malicious separation. Only Alert (90.92%) and Quarantine (9.08%) were observed as policy engine actions, and there were no observations of Allow, Restrict, or Block actions; action precision, containment accuracy, and correctness of enforcement were not measured. Finally, the study results demonstrate the feasibility of the multi-stage IoT security (MSES), and reveal the obvious drawbacks to rare class recognition, unknown-threat precision, trust state coverage, and policy actions. The specific methodological contributions are: (M1) post-classification TARP: topology is separated from the classifier; (M2) confidence-gated OR-fusion: known-traffic screening for unknown traffic; and (M3) source-entity reinterpretation of user trust: traffic-only datasets.