Realistic and balanced automated threat emulation

Cyber defence exercises involve subjecting security analysts to live cyber threats in a safe environment, and is a common proactive method to increase security posture. As the design and execution of cyber threats generally is costly, researchers and practitioners have developed threat emulators that automate cyber threats without the need for human intervention. The ability of these emulators to produce threats useful for cyber defence exercises is, however, uncertain. This paper presents an evaluation of the automated threat emulator Lore using data collected from three cyber defence exercises. During the exercises, Lore and human threat agents (often called the “red” team) subjected 132 network security analysts (often called the “blue” team) to various threats such as software exploits and shell commands. Six hypotheses related to how the actions by human red teams and Lore were perceived and managed by the security analysts were examined. Evaluations were made by studying the subjective judgments of the analysts as well as by comparing the objective ground truth to their submitted incident reports. The results show that the security analysts could not tell the difference between the actions made by the human red team and those made by Lore, and that their performance was similar regardless of the source of the threats. • Automated threat emulators can provide realistic and balanced cyber defence exercises. • The tool Lore provides an as realistic and balanced experience a human red team. • Three exercises involving 132 network security analysts were studied. • Evaluations were made using incident reports and questionnaires.