Enhancing Intrusion Detection System Performance Using Reinforcement Learning : A Fairness-Aware Comparative Study on NSL-KDD and CICIDS2017

Conventional Intrusion Detection Systems (IDS) often fail to generalize in dynamic network environments, facing challenges with evolving attack patterns and class imbalance. This study aims to evaluate and compare the effectiveness of three Reinforcement Learning (RL) paradigms to enhance IDS adaptability and accuracy against these challenges. This research employs a comparative experimental design, implementing Q-Learning, Deep Q-Networks (DQN), and Proximal Policy Optimization (PPO). These algorithms were systematically evaluated using the NSL-KDD and CICIDS2017 benchmark datasets to represent both legacy and modern network traffic. A fairness-aware evaluation framework was applied, prioritizing the Matthews Correlation Coefficient (MCC) as a primary metric alongside accuracy to ensure robust performance assessment against skewed class distributions. Experimental results demonstrate that PPO significantly outperforms value-based algorithms such as Q-Learning and DQN. On the high-dimensional CICIDS2017 dataset, PPO achieved the highest detection accuracy (96.3%) and MCC (0.913). Confusion matrix analyses confirmed PPO’s capability to simultaneously minimize false positives and false negatives. Conversely, Q-Learning exhibited poor generalization on complex data, while DQN showed improved performance due to deep value approximation but remained less stable than PPO. These findings imply that policy-gradient methods like PPO are superior for real-world IDS deployments where scalability, adaptability, and low error rates are critical. Theoretically, the results suggest that stochastic policy optimization handles complex, continuous state spaces more effectively than traditional value-estimation approaches. This study contributes a rigorous head-to-head comparative analysis of RL algorithms across multiple standard datasets using fairness-aware metrics. It bridges the research gap found in previous studies that often evaluated algorithms in isolation or relied on accuracy metrics that can be misleading in imbalanced security contexts.