What Do Capture-the-Flag Competitions Actually Measure? A Psychometric and Competency-Coverage Analysis of Five Editions of a National Cybersecurity Event

Capture-the-flag (CTF) competitions are widely used for cybersecurity training and informal assessment, yet their measurement properties are rarely reported. We evaluated five editions of a national CTF event held between 2018 and 2025 using 94 challenges, 58 teams, 1393 submission attempts, and a 2025 participant survey (n = 38). KR-20 point estimates ranged from 0.72 to 0.82, but were imprecise except in the largest edition; sensitivity analyses treating unattempted items as missing produced weaker, assumption-dependent estimates. Across the corpus, 34% of items had no response variance, and 22% were never attempted. Author-assigned points tracked observed difficulty only weakly (pooled ρ = −0.40, 95% CI [−0.57, −0.19]). Across four coding procedures, technical knowledge was prominent (78–94 items), analytical skills varied substantially (35–70), and ethics, communication, and risk management were absent. A degree-preserving null reproduced an apparent breadth–performance association, indicating a mechanical artifact. Submission precision increased from approximately 0.15 in 2018 to 0.69 in 2025 (Jonckheere–Terpstra z = 6.15, p < 0.001). Overall, reliability alone does not establish validity: CTF scores provide useful but incomplete evidence for broad inferences about cybersecurity competence, particularly with small samples and selective item engagement.