Go Phish - Detecting Phishing Attacks with Prompt Engineered Large Language Models as Classification Tools

This thesis explores the feasibility of using Large Language Models (LLMs) to detect phishing attempts using large, publicly available data sets of emails, and focusing on quantitative measures of success rates. The study compares the accuracy of LLMs to that of state-of-the-art methods, such as machine learning models. This is done by employing prompt engineering to different degrees using state-ofthe-art, commercial LLMs to evaluate the success rate of correctly classifying phishing emails. Since LLMs are developing rapidly, understanding the performance of LLMs as tools to detect phishing attempts still requires further research, which makes up the problem statement of the thesis. By answering the research question: "How do current prompt-engineered Large Language Models perform compared to traditional Machine Learning models in phishing email detection?", the findings presented that today’s Large Language Models performed similarly to Machine Learning (ML) models. A F1 measure of 0.9793 was achieved by an LLM, compared to a 0.9891 value achieved by the comparative ML model. The value of revising prompts based on how the LLMs initially react was also proven to be effective, finding that a minor revision led to a 32 percentage point increase in performance. The thesis presents valuable information on the viability of using LLMs for cyber security measures, with the potential of being further explored with the use of more resources, financially, technically, and in terms of time.