Design and evaluation of an Autonomous Cyber Defence agent using DRL and an augmented LLM

In this paper, we design and evaluate an Autonomous Cyber Defence (ACD) agent to monitor and act within critical network segments connected to untrusted infrastructure hosting active adversaries. We assume that modern network segments use software-defined controllers with the means to host ACD agents and other cybersecurity tools that implement hybrid AI models. Our agent uses a hybrid AI architecture that integrates deep reinforcement learning (DRL), augmented Large Language Models (LLMs), and rule-based systems. This architecture can be implemented in software-defined network controllers, enabling automated defensive actions such as monitoring, analysis, decoy deployment, service removal, and recovery. A core contribution of our work is the construction of three cybersecurity knowledge graphs that organise and map data from network logs, open source Cyber Threat Intelligence (CTI) reports, and vulnerability frameworks. These graphs enable automatic mapping of Common Vulnerabilities and Exposures (CVEs) to offensive tactics and techniques defined in the MITRE ATT&CK framework using Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT) models. Our experimental evaluation of the knowledge graphs shows that BERT-based models perform better, with precision (83.02%), recall (75.92%), and macro F1 scores (58.70%) significantly outperforming GPT models. The ACD agent was evaluated in a Cyber Operations Research (ACO) gym against eleven DRL models, including Proximal Policy Optimisation (PPO), Hierarchical PPO, and ensembles under two different attacker strategies. The results show that our ACD agent outperformed baseline implementations, with its DRL models effectively mitigating attacks and recovering compromised systems. In addition, we implemented and evaluated a chatbot using Retrieval-Augmented Generation (RAG) and a prompting agent augmented with the CTI reports represented in the cybersecurity knowledge graphs. The chatbot achieved high scores on generation metrics such as relevance (0.85), faithfulness (0.83), and semantic similarity (0.88), as well as retrieval metrics such as contextual precision (0.91). The experimental results suggest that the integration of hybrid AI systems with knowledge graphs can enable the automation and improve the precision of cyber defence operations, and also provide a robust interface for cybersecurity experts to interpret and respond to advanced cybersecurity threats. • The motivation for the present investigation comes from the increasing complexity and sophistication of cyber threats targeting critical network segments connected to untrusted infrastructure, such as commercial Internet using wired or wireless links. As a result, the scientific community has been introducing autonomous cyber defence (ACD) agents to enforce counter measures to these cyber threats. • Design of a hybrid Artificial Intelligence (AI) architecture integrating an Autonomous Cyber Defense (ACD) agent using Deep Reinforcement Learning (DRL) with a chatbot using a Retrieval Augmented Generation (RAG) architecture and cybersecurity knowledge graphs. Our Autonomous Cyber Defense (ACD) agent has an action space composed of 14 actions to monitor, analyse, deploy a decoy replacing a real service, remove malicious processes, and restore the entire systems from a backup. • Design and implementation of three knowledge graphs that combine information about the network infrastructure and open source Cyber Threat Intelligence (CTI) reports. This includes an architecture for automatically mapping Common Vulnerabilities and Exposures (CVE) to the set of offensive tactics and techniques defined in the MITRE ATT&CK framework, using generative models based on Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT). Our solution was evaluated in a comparative performance analysis using three Bidirectional Encoder Representations from Transformers (BERT)-based models (CyBERT, SecBERT and SecRoBERTa) and two versions of Generative Pre-trained Transformer (GPT) (GPT-4 and GPT-4o). • Comparative evaluation of an ACD agent acting on the cybersecurity knowledge graph against 11 Deep Reinforcement Learning (DRL) algorithms, Proximal Policy Optimization (PPO) (baseline for the comparison), hierarchical Proximal Policy Optimization (PPO) and ensemble DRL, in a gym environment hosting red agents using two attack strategies. Experimental results indicate that our solution using PPO and cybersecurity knowledge graphs outperforms the other DRL algorithms. • Implementation and performance evaluation of a chatbot (ReAct+Retrieval Augmented Generation (RAG)) using a pre-trained Large Language Model (LLM) to interface with human cybersecurity experts. Our solution using ReAct and RAG was evaluated using both generation (relevancy, faithfulness and semantic similarity) and retrieval (context precision, context recall, and F1 score) metrics in a comparative analysis against a zero-shot implementation (pre-trained Large Language Model (LLM) without augmentation).