Automatic Scoring of Cognition Drawings - Improving prediction accuracy through curriculum learning

Dementia and related disorders are a major public health concern as they are among the leading causes of death, disability and dependency in the global elderly population. There is currently no known cure for the disease, making the identification and management of risk factors one of the few promising approaches to address this emerging crisis. One way to detect early signs of cognitive decline is through cognitive testing (e.g., the Montreal Cognitive Assessment, MoCA), which includes drawing tasks such as cubes or clocks to assess the visuospatial domain of cognitive function. Recently, this type of test has been used in several large-scale survey studies, such as SHARE, the Survey of Health, Ageing and Retirement in Europe (https://share-eric.eu/). While these tests are adapted from clinical cognitive assessments, in this setting they are used to make population-based estimates, such as the prevalence of dementia symptoms, rather than individual diagnoses. This will allow researchers to look for patterns in early cognitive decline and hopefully identify factors that can slow or delay its progression. SHARE includes tests from the Addenbrooke’s Cognitive Examination III (see Wagner & Douhou, 2021, https://share-eric.eu/fileadmin/user_upload/Methodology_Volumes/SHARE_Methodenband_WEB_Wave8_MFRB.pdf), but unlike the clinical setting, scoring of the drawings is not done by trained clinicians, but by the regular face-to-face interviewers during fieldwork, posing a potential risk to data quality. In a proof of concept (Bethmann et al, 2023, https://doi.org/10.48550/arXiv.2312.16887) based on approximately 2,000 cube drawings from the German SHARE sub-study, we re-scored the drawings using multiple raters and a subsequent arbitration round to arrive at a preliminary ‚ground truth’ score. In comparison, interviewer scoring accuracy was only around 75%, with ‚partially correct‘ or ‚incorrect‘ cubes being particularly difficult to score. We then trained several different deep learning models (AlexNet, VGG, ResNet, ConvNeXt) on the ‚ground truth‘ data, which resulted in a significantly better scoring accuracy of around 85% for the best models (ConvNeXt). While these results are promising, they are not yet sufficient for production use. We are therefore pursuing several avenues to improve scoring accuracy and hence data quality: We have collected approximately 55,000 additional recording booklets with cognition drawings from other SHARE countries, which will be scanned and then pre-processed. We expect that training our deep learning models on this training dataset will significantly improve scoring accuracy due to its larger size alone. At the same time, we want to improve the training procedure by using an approach called ‚curriculum learning‘ (cf. Bengio et al., 2009, https://doi.org/10.1145/1553374.1553380), which is a method to gradually increase the complexity of the data samples during the training process, similar to the way humans learn. We want to evaluate whether and to what extent this helps to increase the prediction accuracy of our models.