An Automatic Scoring Method for Responses to the Hayling Test

Timely and objective assessment of executive functions is critical for diagnosing and monitoring cognitive disorders, yet conventional scoring of Hayling Test responses remains subjective and prone to inter-rater variability. To address this challenge, we introduce an end-to-end automated scoring pipeline using state-of-the-art natural language processing (NLP). We systematically benchmark both static embeddings FastText, a range of transformer-based encoders, and recent large language models (LLMs) including Phi-3-mini, Zephyr-7B, XGLM-7.5B, Qwen1.5-7B, DeepSeek-V2, Llama-2-7B, and others on a corpus of 2,414 manually annotated French responses. Each response is classified into three penalty levels: 0 (correct inhibition), 1 (partial inhibition error), and 3 (failure to inhibit). After fine-tuning, the cross-lingual encoder LaBSE achieves the best overall performance, with a Macro-<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{F}_{1}$</tex> of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{0. 8 0}$</tex>, Balanced Accuracy of 0.79, and overall Accuracy of 0.83, outperforming all baselines and LLMs. Among LLMs, Phi-3-mini achieves the highest scores (Macro- <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{F}_{1} \mathbf{0. 5 7}$</tex>, ACC 0.63), yet remains below supervised approaches. Our automated approach standardizes test scoring, reduces subjective bias, and enables nearinstantaneous, scalable neuropsychological assessments paving the way for more reliable and efficient clinical and research workflows. The code supporting this work is publicly available at: https://github.com/insafhamdi/AI-hayling