Multi-Objective and deep Q-Learning for countermeasure selection in 5G intrusion response systems

Network connectivity exposes network infrastructure and assets to vulnerabilities exploitable by attackers. Safeguarding these assets necessitates implementing security countermeasures. However, deploying countermeasures incurs various costs, including preparation and deployment time. Therefore, an Intrusion Response System (IRS) must consider both security and Quality of Service (QoS) costs when dynamically selecting countermeasures to address detected attacks. To address this challenge, we introduce a joint Security-vs-QoS optimization problem akin to the Weighted Set Cover Problem (WSCP), which is NP-complete. We propose two learning-based solutions leveraging Multi-Objective Reinforcement Learning and Deep Q-learning to navigate the security and QoS cost trade-off. Through extensive simulations under diverse settings, we validate the performance of our proposed solution, compare it with benchmark methods, and evaluate it using a project-derived 5G cybersecurity dataset.