High-stakes psychomotor ability assessment: a military selection case study of practice effects in airplane tracking tasks

Aviation selection tests are high-stakes assessments designed to identify candidates capable of succeeding in demanding flight environments. Most branches of the US military incorporate both content-based and process-based assessments to evaluate prior knowledge and reasoning ability, respectively. A challenge with high-stakes process tests is that their validity requires participant naivety, which is increasingly difficult to maintain in the modern internet era. As such, these high-stakes tests must be continuously evaluated to ensure the most valid, reliable, and cost-effective selection procedures are employed. To address this, we examined practice effects in the psychomotor airplane tracking tasks of the Navy's Aviation Selection Test Battery (ASTB). We had 146 Naval Flight Students and 119 enlisted Sailors perform the ASTB's psychomotor battery six times across two days. Results revealed large practice effects, shifting in rank ordering of individuals across attempts, and that psychomotor performance had not stabilized even by the sixth attempt. Prior action video gaming and flight simulator gaming experience correlated with psychomotor performance, with some evidence that improvements were related, albeit weakly, to either gaming experience or initial performance. Finally, correlations between psychomotor performance and eventual naval flight training scores were stable across the six attempts, but simulations indicated predictive validity can range widely if participants have differing levels of practice from one another. Overall, these findings indicate that the psychomotor component of the ASTB is a valuable inclusion to the Naval Flight School selection process but also could benefit from further refinement.