Organizing and Augmenting Cybersecurity Knowledge Using Generative AI

With the rapid advancements in Artificial Intelligence (AI), cybersecurity threats are becoming increasingly sophisticated. Static, human-curated approaches such as MITRE ATT&CK and CAPEC are often insufficient for companies to implement effective countermeasures, and security experts frequently face challenges in integrating these frameworks into their specific architectures. In this paper, we explore the application of generative AI to organize and enhance cybersecurity knowledge bases. We leverage multiple state-of-the-art Large Language Models (LLMs) to augment an initial dataset of attackmitigation pairs, generating new countermeasures along with their prioritized execution order in a reasonable time frame. To evaluate the generated responses, we employ the LLM-as-aJudge technique. Evaluations with Claude 3.5, DeepSeek R1, and human oversight show that top-performing models on general benchmarks also perform well in this specialized task, with more than 70% of responses rated 4 and 5 out of 5 for correctness, and the weaker models tend to perform the worst. Additionally, we discuss the challenges of running LLMs with a larger number of parameters on less powerful hardware, where the performance of such models can degrade significantly, even performing worse than their smaller counterparts. In such cases, advanced prompt engineering becomes necessary to improve results. Our code and dataset are publicly available at “https://github.com/martel-innovate/HORSE-GenAI-CKB”