Enhanced Intrusion Detection Using Deep Q-Network with Priority Experience Replay and Multi-Agent Reinforcement Learning

Intrusion Detection Systems (IDS) play a critical role in cybersecurity by monitoring network traffic and detecting potential threats.Traditional IDS approaches face challenges in detecting sophisticated and evolving cyberattacks while maintaining low false positive rates.This study introduces a new framework utilizing Deep Q-Networks (DQN) with Priority Experience Replay (PER) and Multi-Agent Feature Selection (MAFS) to enhance IDS effectiveness.By integrating Graph Convolutional Networks (GCN) for feature representation, the system improves the detection of complex network traffic patterns, ensuring adaptability and scalability in real-time environments.The methodology employs a combination of GCN for high-level feature extraction, reinforcement learning-based MAFS to select relevant features dynamically, and PER to prioritize critical samples with high temporal difference errors.These prioritized samples enable the DQN to learn from significant but infrequent network patterns, increasing detection accuracy.The framework balances model accuracy and computational efficiency using a reward function that optimizes feature subset size and classification performance.Results show that the proposed system, tested on CSE-CIC-IDS2018 and NSL-KDD datasets, outperforms traditional models like Decision Trees and baseline DQN.We propose a model that obtained 97.8% accuracy and 0.97 F1 score for CSE CIC IDS2018 and 99.5 % accuracy and 0.993 F1 score for NSL KDD.Additionally, the proposed model significantly reduces false positives and false negatives, addressing alert fatigue and improving detection reliability.In conclusion, the proposed IDS framework combines advanced reinforcement learning techniques with feature selection to provide a dynamic, scalable, and highly accurate intrusion detection system capable of adapting to evolving cyber threats.