LLM-Assisted Annotation for Spanish Medical Corpus Construction: An Empirical Evaluation

Named entity annotation in clinical text is a costly and time-intensive task, particularly for low-resource languages such as Spanish. This article introduces CLINFEM, an XML-annotated corpus comprising 1,200 Spanish medical reports from two regions of Spain. CLINFEM was created using a thorough human-in-the-loop annotation process combining large language model pre-annotation and systematic expert review. It is imperative to acknowledge that each of the 1,200 medical reports underwent meticulous review and correction by a trained human expert. Large language model pre-annotation was employed solely for the purpose of saving time and was never intended to serve as a substitute for manual annotation. Consequently, the entire CLINFEM corpus is a human-validated, gold-standard resource. GPT-4o and Gemini 1.5 Pro were evaluated as pre-annotators using a custom scheme comprising eight clinical entity types and one relational category. GPT-4o achieved a Cohen's κ of 0.73 against human annotation, while Gemini 1.5 Pro obtained κ = 0.41. Human expert review yielded an overall inter-annotator agreement of κ = 0.869 (F1 = 0.793), with a gold-standard subset agreement of κ = 0.9894. This approach reduced the overall annotation effort by around 74% compared to a fully manual approach (81% for the bulk phase alone), saving over 1,000 hours. Error rates varied by medical speciality, with oncology requiring the highest correction rate (94.3% of reports). These results demonstrate that using large language models to assist annotation is a viable and efficient strategy for constructing clinical corpora in Spanish, while also highlighting the continued necessity of human oversight.

Authors

Institutions

Publication Details

Journal
Journal of Computer-Assisted Linguistic Research
Published
2026-11-18
DOI
https://doi.org/10.4995/jclr.2026.26224
Primary Topic
Biomedical Text Mining and Ontologies
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

LLM-Assisted Annotation for Spanish Medical Corpus Construction: An Empirical Evaluation

Borja Navarro-Colorado, O. Sanchez
3 links
Journal of Computer-Assisted Linguistic Research
Biomedical Text Mining and Ontologies
article

LLM-Assisted Annotation for Spanish Medical Corpus Construction: An Empirical Evaluation

Borja Navarro-Colorado, O. Sanchez
article en

Abstract

Named entity annotation in clinical text is a costly and time-intensive task, particularly for low-resource languages such as Spanish. This article introduces CLINFEM, an XML-annotated corpus comprising 1,200 Spanish medical reports from two regions of Spain. CLINFEM was created using a thorough human-in-the-loop annotation process combining large language model pre-annotation and systematic expert review. It is imperative to acknowledge that each of the 1,200 medical reports underwent meticulous review and correction by a trained human expert. Large language model pre-annotation was employed solely for the purpose of saving time and was never intended to serve as a substitute for manual annotation. Consequently, the entire CLINFEM corpus is a human-validated, gold-standard resource. GPT-4o and Gemini 1.5 Pro were evaluated as pre-annotators using a custom scheme comprising eight clinical entity types and one relational category. GPT-4o achieved a Cohen's κ of 0.73 against human annotation, while Gemini 1.5 Pro obtained κ = 0.41. Human expert review yielded an overall inter-annotator agreement of κ = 0.869 (F1 = 0.793), with a gold-standard subset agreement of κ = 0.9894. This approach reduced the overall annotation effort by around 74% compared to a fully manual approach (81% for the bulk phase alone), saving over 1,000 hours. Error rates varied by medical speciality, with oncology requiring the highest correction rate (94.3% of reports). These results demonstrate that using large language models to assist annotation is a viable and efficient strategy for constructing clinical corpora in Spanish, while also highlighting the continued necessity of human oversight.

Journal of Computer-Assisted Linguistic ResearchVol. 10
University of Alicante (ES)
Universidad de Alicante, Generalitat Valenciana, Agencia Estatal de Investigación
Good health and well-being
Biomedical Text Mining and Ontologies
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon