Machine and deep learning models do not distinguish concussion history from clinical and robotic assessments in youth ice hockey players

Concussion is common in youth ice hockey players. Concerns about the long-term effects on the developing brain highlight a need for objective assessment tools. In this study, we used standardized robotic tasks and clinical assessments to evaluate youth ice hockey players who were clinically asymptomatic and playing hockey at the time of testing. These features were used to train machine learning and deep learning models to classify concussion history. Players in the Under-13 age group were recruited from a 5-year longitudinal youth ice hockey study (Safe2Play) to complete clinical assessments (pre-season questionnaire and SCAT3) and five upper limb robotic tasks (Kinarm). In total, 91 parameters were analyzed. First, we quantified out-of-range performance in individual robotic task parameters by determining whether participants with concussion history fell outside performance ranges derived from participants without concussion history. We then applied five machine learning models and one deep learning model, each with hyperparameter tuning, to classify concussion history. Inputs included robotic task parameters and clinical assessments, and classification performance was evaluated with 10-fold cross-validation. Performance was assessed using accuracy, precision, recall, F1 score, and the area under (AUC) receiver operating characteristic (ROC) and precision-recall (PR) curves. Three hundred and twenty pediatric ice hockey players participated in this study (101 with and 219 without history of concussion; ages 10-12, median days since concussion = 702, range = 33-3610). Across individual robotic parameters, 1.0–14.9% of participants with concussion history fell outside cut-off limits derived from 95% of the performance range in participants without concussion history. At the group level, robotic parameters showed small effects (absolute Cohen’s d = 0.01–0.26) and limited discriminability (ROC AUC = 0.41–0.57, PR AUC = 0.26-0.39). Clinical variables also showed small group-level effects (absolute Cliff’s δ = 0.01–0.12) and limited discriminability (ROC AUC = 0.47–0.56, PR AUC = 0.03-0.36). The mean classification accuracy across models was 64.1 ± 3.1%, with low precision (36.8 ± 4.7%), recall (22.5 ± 11.9%), and F1 scores (25.9 ± 10.3%). Overall discrimination was near chance (ROC AUC = 0.53 ± 0.03; PR AUC = 0.41 ± 0.04). Model performance was similar across algorithms, and features contributing to classification varied across models. Youth hockey players with concussion history were clinically asymptomatic and showed small group-level differences, limited discriminability on individual clinical and robotic parameters, and few out-of-range values on robotic parameters. Machine learning and deep learning models demonstrated poor classification performance using the multivariate clinical and robotic features. These results indicate that remote concussion history was not associated with pronounced differences in clinical or robotic assessments in this cohort of asymptomatic youth ice hockey players.