Artificial intelligence-assisted assessment of the forward roll in the Fundamental Motor Skills in Sport (FUS) test: agreement with expert consensus, inter-rater reliability, and rater workload

Low proficiency in fundamental motor skills among children highlights the need for valid, feasible, and reliable assessment methods to support instruction and population-level monitoring. This study examined agreement between artificial intelligence (AI)-generated forward roll scores and expert consensus ratings and whether criterion-level AI support was associated with closer agreement between human ratings and expert consensus, higher inter-rater reliability, and lower perceived workload within the Fundamental Motor Skills in Sport (FUS) test. Scores from an AI model were compared with expert consensus ratings in 66 forward roll videos spanning the full 0–5 total-score range using Cohen’s kappa and quadratic weighted kappa. In a counterbalanced experiment, 20 raters from Malta and Poland assessed two independent sets of 22 videos, one with and one without criterion-level AI support. Human rating error and agreement were evaluated against a separate blinded expert consensus, while inter-rater reliability was estimated descriptively. Perceived workload was assessed after each condition using the NASA Task Load Index. AI-generated criterion-level scores showed substantial to almost perfect agreement with expert consensus (kappa 0.73–0.85). Quadratic weighted kappa between AI-generated and expert-consensus total scores was 0.95 (95% CI 0.92–0.97). Human ratings with AI support more closely matched the separate blinded expert consensus: mean absolute error was 0.68 (95% CI 0.53–0.85) versus 0.99 (95% CI 0.85–1.13), and quadratic weighted kappa was 0.79 (95% CI 0.65–0.87) versus 0.60 (95% CI 0.44–0.70). Error was lower for 19 of 20 raters (Wilcoxon p < 0.001, conditional on sampled videos). Descriptive ICC(2,1) point estimates were higher with AI support in all four country-by-video-set comparisons. After Holm correction, only perceived effort was significantly lower with AI support ( p = 0.045). Under standardized video conditions, the AI system showed strong agreement with expert consensus. Human ratings made with AI support also more closely matched a separate blinded consensus, and raters reported less effort. These task-specific findings support further evaluation of criterion-level AI recommendations as a decision aid for human-supervised forward roll scoring. The rater findings concern the participating master’s students. Extension to practicing teachers, other age groups, other skills, or routine school settings requires separate validation.

Authors

Institutions

Publication Details

Journal
BMC Sports Science Medicine and Rehabilitation
Published
2026-11-14
DOI
https://doi.org/10.1186/s13102-026-02232-4
Primary Topic
Children's Physical and Motor Development
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Artificial intelligence-assisted assessment of the forward roll in the Fundamental Motor Skills in Sport (FUS) test: agreement with expert consensus, inter-rater reliability, and rater workload

Beata Makaruk, Hubert Makaruk, Andrew Decelis, Jerzy Sadowski et al.
1 links
BMC Sports Science Medicine and Rehabilitation
Children's Physical and Motor Development
article

Artificial intelligence-assisted assessment of the forward roll in the Fundamental Motor Skills in Sport (FUS) test: agreement with expert consensus, inter-rater reliability, and rater workload

Beata Makaruk, Hubert Makaruk, Andrew Decelis, Jerzy Sadowski, Paweł Tomaszewski
article en

Abstract

Low proficiency in fundamental motor skills among children highlights the need for valid, feasible, and reliable assessment methods to support instruction and population-level monitoring. This study examined agreement between artificial intelligence (AI)-generated forward roll scores and expert consensus ratings and whether criterion-level AI support was associated with closer agreement between human ratings and expert consensus, higher inter-rater reliability, and lower perceived workload within the Fundamental Motor Skills in Sport (FUS) test. Scores from an AI model were compared with expert consensus ratings in 66 forward roll videos spanning the full 0–5 total-score range using Cohen’s kappa and quadratic weighted kappa. In a counterbalanced experiment, 20 raters from Malta and Poland assessed two independent sets of 22 videos, one with and one without criterion-level AI support. Human rating error and agreement were evaluated against a separate blinded expert consensus, while inter-rater reliability was estimated descriptively. Perceived workload was assessed after each condition using the NASA Task Load Index. AI-generated criterion-level scores showed substantial to almost perfect agreement with expert consensus (kappa 0.73–0.85). Quadratic weighted kappa between AI-generated and expert-consensus total scores was 0.95 (95% CI 0.92–0.97). Human ratings with AI support more closely matched the separate blinded expert consensus: mean absolute error was 0.68 (95% CI 0.53–0.85) versus 0.99 (95% CI 0.85–1.13), and quadratic weighted kappa was 0.79 (95% CI 0.65–0.87) versus 0.60 (95% CI 0.44–0.70). Error was lower for 19 of 20 raters (Wilcoxon p < 0.001, conditional on sampled videos). Descriptive ICC(2,1) point estimates were higher with AI support in all four country-by-video-set comparisons. After Holm correction, only perceived effort was significantly lower with AI support ( p = 0.045). Under standardized video conditions, the AI system showed strong agreement with expert consensus. Human ratings made with AI support also more closely matched a separate blinded consensus, and raters reported less effort. These task-specific findings support further evaluation of criterion-level AI recommendations as a decision aid for human-supervised forward roll scoring. The rater findings concern the participating master’s students. Extension to practicing teachers, other age groups, other skills, or routine school settings requires separate validation.

BMC Sports Science Medicine and Rehabilitation
University of Malta (MT), Józef Piłsudski University of Physical Education in Warsaw (PL)
Good health and well-being
Children's Physical and Motor Development
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon