A Novel Dual‐Path Feature Engineering Framework for Scalable and Accurate Intrusion Detection

ABSTRACT The rapid proliferation of IoT devices and networked services has intensified cybersecurity challenges, driven by the diversity of protocols, massive traffic volumes, and the increasing sophistication of cyber threats. These threats undermine the confidentiality, integrity, and availability of modern networks, while traditional intrusion detection systems (IDS) often struggle to balance accuracy, computational efficiency, and adaptability. To address these limitations, we propose a novel AI‐driven IDS framework that combines advanced feature selection and compression strategies. The framework employs label encoding to enhance data representation, followed by Extra Trees‐based feature selection to identify the most informative attributes. Unlike conventional approaches that discard uncorrelated features, our methods, ETC‐DFC (Extra Trees Classifier with Deep Feature Compression) and ETC‐SFC (Extra Trees Classifier with stacked Feature Compression), retain and compress both correlated and uncorrelated features into compact embeddings, preserving critical information while reducing storage requirements and improving computational efficiency. Extensive experiments conducted on the CIC‐IDS2017, CSE‐CIC‐IDS2018, and CIC‐IoT 2023 datasets demonstrate the effectiveness of the proposed approaches, with XGBoost achieving 99.98% and 99.97% accuracy for binary and multiclass classification on CIC‐IDS2017, 98.72% (ETC‐SFC) and 98.20% (ETC‐DFC) on CSE‐CIC‐IDS2018, and up to 99.98% on CIC‐IoT 2023. Further validation on the Edge‐IIoTset and ICS‐Flow datasets, which simulate realistic IT, IoT, and industrial control traffic, demonstrates the potential of the proposed framework for scalability, robustness, and practical deployment.