Local language models for Spanish L2 writing assessment: human baselines, RAG, and fine-tuning

Abstract This study investigates the use of local Large Language Models (LLMs): Gemma, LLama, Mistral Small, and Phi, in assessing Spanish second language (L2) writing. While commercial models like ChatGPT have already been used for this task, they present privacy risks and reproducibility challenges. To evaluate local alternatives to ChatGPT, human raters were used to assess a third-semester assignment, evaluating 100 paragraphs to provide a baseline of comparison for the local LLMs’ assessment. Results showed varying performance: while some models approached the human baseline; others were inaccurate and inconsistent. On average, they performed better using a holistic rubric than an analytic one. A third experiment, averaging the top 3 models: Llama3.3:70b, Gemma2:9b, Mistral-Small:22b, achieved the highest agreement ( r = 0.88), surpassing inter-human rating. A separate experiment testing RAG and fine-tuning techniques on the smallest model: Llama3.1:8b, showed no significant improvements with RAG but substantial gains in accuracy and consistency with fine tuning. Findings suggest that local LLMs show promise for Spanish L2 writing assessment. Local LLMs are preferable to proprietary ones because they protect privacy and reduce risks of data exposure, they may better support replicable workflows and may offer practical advantages for long term adoption.