Agentic hybrid intrusion detection system with autoencoder-based zero-day detection and transformer–tree ensemble fusion

The rapid growth of Internet of Things and Industrial Internet of Things environments has significantly increased the risk of sophisticated cyberattacks, making accurate and robust intrusion detection a critical security requirement. Traditional intrusion detection systems depend on single-model that struggle to simultaneously detect known attack patterns and unseen zero-day threats that shows in limited generalization and reduced detection performance. This study proposes a novel Agentic Artificial Intelligence-based Intrusion Detection System that integrates anomaly detection, deep learning, ensemble learning, and agent-based decision fusion within a single model. The proposed model used an Autoencoder-based anomaly detection agent to identify zero-day attacks through reconstruction error analysis, followed by two complementary classification agents: a TabTransformer for contextual representation learning from tabular network traffic data and an XGBoost classifier for robust gradient-boosted classification. The outputs of agents were combined using Dempster–Shafer evidence theory and refined through a confidence-aware meta-agent with class-specific threshold tuning. The SHAP and LIME explainability techniques used to enhance model interpretability. The performance of model measured across Edge-IIoT intrusion dataset that shows detection capability to obtained AUC of 99.0% across multiple attack classes and statistically significant improvements over baseline models (p < 0.01). The results confirm that the proposed Agentic IDS model improved accuracy, robustness, explainability, and generalization for real-world IoT and IIoT cybersecurity applications.