Evaluating GPT-4o as a Cyberattack Simulator: Perspectives on AI and Human Decision-Making

Large language models (LLMs) such as GPT-4o are gaining attention for their ability to mimic human behaviors, but their use in replicating hacking strategies in the field of cybersecurity is still not fully examined. This investigation assesses GPT-4o as a simulator for cyberattacks, focusing on an important area in utilizing AI to examine the decision-making processes of adversaries. Through the HackIT simulation platform, GPT-4o's decision-making was compared against 84 human participants across bus and hybrid network topologies, under varying configurations of temperature (0.5, 1, 1.5) and top-k sampling (2, 3, 4). The results showed that GPT-4o did as well as human participants. It used an average of 32 systems in bus topologies and 28 systems in mixed topologies, and the mean squared errors (MSE) ranged from 0.03 to 1.44 in each case. In particular, the AI model was better at adapting to linear configurations and was very good at recognizing real systems, sometimes surpassing human performance. These results show that GPT-4o has a lot of promise as a defense tool for predicting and evaluating, giving us new information about how attackers work and where networks are weak. To make LLM-driven defense solutions even better, more study should be done on dynamic and adaptive attack situations.