Measuring the unseen: a situational awareness scoring framework for cyber range training

Abstract As cyber-attacks grow in scale, sophistication, and impact, advanced training and evaluation platforms are essential for cybersecurity readiness. Cyber Ranges (CRs) provide controlled, high-fidelity simulation environments in which professionals can practice defensive strategies under realistic threat scenarios. In such environments, success depends not only on technical skill but also on robust Situational Awareness (SA), the cognitive capacity to perceive events, understand their meaning, and project future developments. Although SA is recognised as critical, existing CR evaluation methods typically focus on overall performance metrics (e.g., task completion rates, response accuracy, mission success). Such assessments often fail to capture the deeper, cognitive aspects of SA, limiting the CR’s ability to evaluate decision-making agility and strategic insight. In response, this work presents the SA Scoring System (SASS), a dynamic, multi-dimensional framework designed to quantify SA in live CR environments. SASS decomposes SA into core dimensions, perception, comprehension, projection, and resolution, then monitors trainee behaviour and scenario interactions in real-time. Each dimension is assessed via weighted metrics: timing of responses, sequence and number of reasoning steps, usage and effectiveness of tools and techniques, hints requested, and decision outcomes. These metrics are aggregated into interpretable scores that reflect both aspects, specific performance and overall SA proficiency. In this study, the SASS is validated in a series of cyber-defence exercises involving participants of differing expertise levels. The results demonstrate that SASS scores correlate strongly with expert evaluations and can discriminate between high and low SA performers with high fidelity. SASS enables instructors to pinpoint cognitive weak spots (e.g. delayed perception or faulty projection) and adapt training on the fly. Ultimately, integrating the SASS into CR platforms enriches the evaluation process, promoting not just technical competence but also the situational agility needed to counter modern cyber threats.