Leveraging Large Language Models for Autonomous Cyber Defense: Insights from CAGE-2 Simulations

The ability of large language models (LLMs) to interpret and generate human-like text, along with their inbuilt knowledge and reasoning skills, has opened the door for many new applications. We develop a novel LLM agent as an autonomous cyber defender and benchmark it using the CAGE-2 public challenge which evaluates submitted agents in a simulated network. The agent uses a pre-trained LLM to interpret a text description of the challenge and choose the best defensive action given a network observation. Its performance is on par with the most successful reinforcement learning (RL) agent submitted to CAGE-2 with the advantages of explainability and avoiding training episodes, but with the cost of requiring LLM prompt engineering. Lastly, we test the LLM agent experimentally to explore how LLM temperature and prompt complexity affect performance. We show that higher complexity prompts significantly improve performance, while LLM temperature has only a minor effect.