Clinical validation of an Artificial Intelligence software for bone age assessment based on Greulich and Pyle method in a Portuguese paediatric cohort

The implementation of an Artificial Intelligence (AI) software for bone age (BA) assessment in clinical practice requires external independent validation. The current study aimed to analyse the performance of AI software (BoneAge™, Gleamer) in BA estimation in a Portuguese paediatric cohort. All frontal hand radiographs acquired between January 2022 and December 2023 in the imaging department of a secondary level hospital were retrospectively collected. Two expert raters defined the reference standard for true BA in consensus according to Greulich and Pyle atlas. Overall, gender and age-specific results were analysed. The study included 267 radiographs from 229 patients. The AI software evidenced high reliability for BA assessment, with consistent performance (r = 0.98) and strong agreement with the reference standard (mean average error and root mean squared error of 4.9 months). An underestimation bias was shown (mean difference of -2.55 months), which was also demonstrated in both sex (more evident in males), and in the non-adolescents group. Bias was not statistically significant in the adolescents’ subgroup. The AI algorithm presented a sensitivity of 67.6 %, specificity of 96.2 %, positive predictive value of 92.4 %, negative predictive value of 81.4 % and accuracy of 84.6 %. The AI software is reliable for BA prediction with a small underestimation bias. It is highly effective for ruling out healthy cases, but it does not correctly identify all pathological cases. Further prospective multi-centre studies are advised. • External independent validation is required before clinical implementation of Artificial Intelligence (AI) tools. • BoneAge™ showed strong performance in bone age assessment (r = 0.98; MAE and RMSE = 4.9 months). • A small but significant underestimation bias (MD −2.55 months) was found in both sexes and the non-adolescent subgroup. • AI showed 81.4% accuracy, 96.2% specificity, and 67.6% sensitivity, effectively ruling out healthy cases. • Further prospective multi-centre studies that analyze the performance of a human reader augmented by AI are advised.