Generating Phishing Attacks and Novel Detection Algorithms in the Era of Large Language Models

Phishing is a significant cybersecurity threat, with the financial impact of email security breaches and lack of awareness estimated to be between $50-100 billion in 2022. The advent of Large Language Models (LLMs) has further automated and intensified phishing attacks, posing greater challenges for defenders, especially large organizations being targeted by Advanced Persistent Threats (APT) at scale, such as Department of Energy National Labs. This study presents the development of two innovative algorithms. The first algorithm improves the efficacy of phishing attacks, while the second algorithm counteracts and defends against phishing attacks that leverage LLMs. The attack method takes detectable malicious phishing emails and rewrites them using an innovative LLM-based automatic output optimization technique, which includes Reflection and Beam Search, while preserving the original semantic meaning and Indicators Of Compromise (IOC). This approach bypasses most-commonly used institutional security tools, NLP and other LLM phishing detection systems. The results indicate that this attack algorithm increases the success rate of phishing attacks by up to 98%. The defensive algorithm presented in this research is also employed for defensive measures. When the proposed defensive algorithm is applied, it identifies malicious emails with 97% greater accuracy. The research detailed in this paper demonstrates that these algorithm serve dual purposes: one is utilized as an attack mechanism by altering the output, and the other as a defensive measure against phishing attacks by modifying the defensive prompt. Taking these algorithms and implementing them in the Department of Energy Laboratory (DOE) has demonstrated the effectiveness of applying these approaches to real world applications, and has been implemented into large-scale production environments.