A deep learning approach for automated scoring of the Rey–Osterrieth complex figure

Memory deficits are a hallmark of many different neurological and psychiatric conditions. The Rey–Osterrieth complex figure (ROCF) is the state-of-the-art assessment tool for neuropsychologists across the globe to assess the degree of non-verbal visual memory deterioration. To obtain a score, a trained clinician inspects a patient’s ROCF drawing and quantifies deviations from the original figure. This manual procedure is time-consuming, slow and scores vary depending on the clinician’s experience, motivation, and tiredness. Here, we leverage novel deep learning architectures to automatize the rating of memory deficits. For this, we collected more than 20k hand-drawn ROCF drawings from patients with various neurological and psychiatric disorders as well as healthy participants. Unbiased ground truth ROCF scores were obtained from crowdsourced human intelligence. This dataset was used to train and evaluate a multihead convolutional neural network. The model performs highly unbiased as it yielded predictions very close to the ground truth and the error was similarly distributed around zero. The neural network outperforms both online raters and clinicians. The scoring system can reliably identify and accurately score individual figure elements in previously unseen ROCF drawings, which facilitates explainability of the AI-scoring system. To ensure generalizability and clinical utility, the model performance was successfully replicated in a large independent prospective validation study that was pre-registered prior to data collection. Our AI-powered scoring system provides healthcare institutions worldwide with a digital tool to assess objectively, reliably, and time-efficiently the performance in the ROCF test from hand-drawn images.

Authors

Institutions

Publication Details

Journal
eLife
Published
2024-11-28
DOI
https://doi.org/10.7554/elife.96017.3
Citations
2
Primary Topic
Dementia and Cognitive Impairment Research
Type
article
Field-Weighted Citation Impact
0.77

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

A deep learning approach for automated scoring of the Rey–Osterrieth complex figure

Oskar G. Jenni, Tino Zaehle, Ce Zhang, Flavia M. Wehrle et al.
2 citations
eLife
Dementia and Cognitive Impairment Research
0.77
article

A deep learning approach for automated scoring of the Rey–Osterrieth complex figure

Oskar G. Jenni, Tino Zaehle, Ce Zhang, Flavia M. Wehrle, Nicolas Langer, Marius Troendle, Juan Carlos Arango‐Lasprilla, Qianhua Zhao, Jonathan Heitz, Christoph Schultheiß, Lukas Wolf, Diego Rivera, Maurice Weber, Romy Lorenz, Andreas Pedroni, Stephan A. Müller, Federica Scarpina, Bruno Hebling Vieira, Peter Brügger, Dawid Strzelczyk, Rico Leuthold
article en
2 citations

Abstract

Memory deficits are a hallmark of many different neurological and psychiatric conditions. The Rey–Osterrieth complex figure (ROCF) is the state-of-the-art assessment tool for neuropsychologists across the globe to assess the degree of non-verbal visual memory deterioration. To obtain a score, a trained clinician inspects a patient’s ROCF drawing and quantifies deviations from the original figure. This manual procedure is time-consuming, slow and scores vary depending on the clinician’s experience, motivation, and tiredness. Here, we leverage novel deep learning architectures to automatize the rating of memory deficits. For this, we collected more than 20k hand-drawn ROCF drawings from patients with various neurological and psychiatric disorders as well as healthy participants. Unbiased ground truth ROCF scores were obtained from crowdsourced human intelligence. This dataset was used to train and evaluate a multihead convolutional neural network. The model performs highly unbiased as it yielded predictions very close to the ground truth and the error was similarly distributed around zero. The neural network outperforms both online raters and clinicians. The scoring system can reliably identify and accurately score individual figure elements in previously unseen ROCF drawings, which facilitates explainability of the AI-scoring system. To ensure generalizability and clinical utility, the model performance was successfully replicated in a large independent prospective validation study that was pre-registered prior to data collection. Our AI-powered scoring system provides healthcare institutions worldwide with a digital tool to assess objectively, reliably, and time-efficiently the performance in the ROCF test from hand-drawn images.

eLifeVol. 13
Universidad Pública de Navarra (UPNA) (ES), Virginia Commonwealth University (US), University of Zurich (CH), ETH Zurich (CH), Navarre Institute of Health Research (ES), Max Planck Institute for Biological Cybernetics (DE), University Children's Hospital Zurich (CH), Max Planck Institute for Human Cognitive and Brain Sciences (DE), IRCCS Istituto Auxologico Italiano (IT), University Hospital Magdeburg (DE), National Rehabilitation Center (KR), Ospedale San Giuseppe (IT), Child Development Center (QA), Healthy Start (US), Istituti di Ricovero e Cura a Carattere Scientifico (IT), Huashan Hospital (CN), University of Turin (IT)
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung, Staatssekretariat für Bildung, Forschung und Innovation
Openalex Percentile: Top 25%
Dementia and Cognitive Impairment Research
0.77
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.