Design of an Iterative Method Leveraging Deep Q-Networks for Intrusion Detection System Operations

The complexity and the level of sophistication in cyber-attacks that challenge conventional security measures make the need for advanced intrusion detection systems even more compelling. The major challenges in the conventional NIDS include high false-positive rates, poor adaptability to new and emerging attack vectors, and high computational demand, hence degrading effectiveness and scalability. In addressing these limitations, this work proposes a novel framework for deep reinforcement learning (DRL) algorithms in enhancing NIDS capabilities and efficiency. Five DRL techniques are used in the proposed model: DQN, DDPG, PPO, TD3, and SAC. Each of these algorithms is chosen for their specific strengths in handling the dynamic and adversarial nature of network security environments. DQN is used because of its excellence in addressing the discrete action space, making it very suitable for binary decision-making tasks in intrusion detection. DDPG extends the model to continuous action space, enabling the model to respond to complex threat scenarios. PPO is integrated into the model for its stability and sample efficiency essential for resource-constrained settings. TD3 addresses the overestimation bias in DDPG, thus improving the reliability of the generated security alerts. Finally, SAC brings an exploration-centric approach, ensuring robust adaptability to emerging threats. Initial results show a substantial increase in intrusion detection accuracy, reduced false positive rates, and better computational efficiency compared to traditional NIDS approaches when benchmarked. Thus, this research not only advances the state of NIDS but also contributes to scalable and adaptive security solutions necessary for defending against current and future cyber threats. The proposed model had a detection accuracy of 95.8%, which far outperformed the current approaches such as Random Forest with an accuracy of 91.2% and GAN-based approaches with 90.4%, at a false positive rate of just 2.5%. The model had a detection rate of 97.9% against DoS attacks and achieved an F1-score of 0.92 for U2R attacks. The system processed up to 2,400 transactions per second, showing better scalability and real-time applicability.