Can LLM-Embeddings Detect Phishing URLs?

Phishing attacks continue to be a prevalent cyber-security threat that mimics trusted websites to steal sensitive user information. This results in considerable personal and organisational damage. Traditional countermeasures like URL blacklisting and recent machine learning and deep learning models are limited in terms of generalisability. While more recent large language models (LLMs) have been investigated for phishing detection, few have evaluated LLM-based URL embeddings for phishing detection. To address this issue, in this paper, we evaluate three different LLMs using three URL datasets, for phishing detection based on URL embeddings extracted from fine-tuned models. Our evaluations showed Gemma-2B to be the best-performing model with an average F1 score of 0.91 across all test sets. Using the best-performing model we further demonstrate two model implementations: a webpage implementation for users to determine if a URL is phishing or not, and a Gmail API integration to classify which emails are spam.