Design and Evaluation of a Large Language Model-powered Threat Modeling Approach with Applications to AI Security

Threat modeling is an important technique in the field of cybersecurity that helps to systematically identify and mitigate risks in systems. As systems based on artificial intelligence (AI) become increasingly complex, existing approaches to threat modeling often require extensive documentation, which makes threat modeling impractical in early stages of development. This thesis investigates the feasibility of using large language models to support threat modeling for AI systems, with a focus on threat identification and validation with minimal inputs. A comprehensive literature review revealed that existing solutions are primarily based on detailed system documentation and do not offer an approach that works only with minimal inputs such as data-flow diagrams. To address this gap, this thesis presents a prototype that uses Retrieval Augmented Generation to identify and validate AI threats using only data-flow diagrams as input. The system is designed with a two-stage approach that separates the threat identification and validation processes. Furthermore, this thesis investigates different Retrieval Augmented Generation configurations and evaluates the performance of different large language models in these processes. This evaluation is conducted through a number of interviews that provide information on which combinations of Retrieval Augmented Generation configurations and large language models provide the most effective results. This thesis demonstrates that threat modeling based on an large language model approach can be a valuable for identifying and validating security threats in AI systems. In addition, it provides a practical guidance on optimal configurations for such a tool and gives examples for future research in this field.