Using Topic Modeling and LLMs to Recommend CAPEC Attack Patterns: A Comparative Study

As technology becomes more prominent today, the need for cybersecurity increases. Software developers must develop secure software systems. Common Attack Pattern Enumeration and Classification (CAPEC) is a community resource developed by the U.S. Department of Homeland Security as part of the Software Assurance strategic initiative of the Office of Cybersecurity and Communications. The CAPEC repository provides a collection of over 500 attack patterns, which contains information on software vulnerabilities and how they can be exploited using the given attack pattern. With the repository containing so much information, it can be challenging for software developers to identify which attack pattern is most relevant to their project. This project compares three methodologies for recommending relevant attack patterns: topic modeling, text embedding with OpenAI’s GPT-40-mini model, and prompting with the same model. These methods are evaluated based on the relevance of the recommended attack patterns to the software requirement specification project being tested. The CAPEC description and the prerequisites for each attack as criteria. A publicly available software requirement specification (SRS) document will be used to evaluate these three methods. The results showed that the prompting method was the bestperforming method for recommending attack patterns.