The RECUR corpus: annotated German radiology reports for urolithiasis research

Background Advances in clinical NLP depend on high-quality datasets that reflect the complexity of real-world documentation. In response, a fine-grained, relation-annotated corpus for urinary stone diagnostics is introduced. The corpus addresses a gap in German medical NLP resources and provides a reference dataset for developing and evaluating clinical information extraction models. Methods We extracted and de-identified 262 radiology reports and 251 corresponding short findings concerning kidney and ureter stones from the hospital information system of the University Medical Center Freiburg. Using a detailed annotation schema comprising nine entity types and five relation types, three medical doctoral students annotated the corpus in Label Studio, followed by expert adjudication to establish a gold standard. For evaluation, we fine-tuned several German general-domain and medical language models for named entity recognition (NER) and relation extraction (RE) using a fixed train–development–test split, a unified Hugging Face pipeline, and NNI-based hyperparameter tuning. Results The resulting corpus comprises 513 radiology documents with 3,223 annotated entities, dominated by stone- and anatomy-related categories. Inter-annotator agreement was consistently high for both entity and relation annotations (NER F1 = 82.8%; RE F1 = 81.8%), supporting the reliability of the annotation schema. The evaluated models achieved macro F1-scores of up to 89.45% for NER and 99.24% for RE. While NER performance varied across models and fine-grained entity categories, RE performance was consistently high across all evaluated encoders. Conclusion RECUR establishes a fine-grained reference corpus and evaluation setting for automated information extraction from German radiology reports. Its adjudicated annotations, strong inter-annotator agreement, and comprehensive encoder baselines support the systematic evaluation of clinical NER and RE methods.

Authors

Institutions

Publication Details

Journal
BMC Medical Informatics and Decision Making
Published
2026-11-19
DOI
https://doi.org/10.1186/s12911-026-03982-4
Primary Topic
Biomedical Text Mining and Ontologies
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

The RECUR corpus: annotated German radiology reports for urolithiasis research

Tabea Walther, Luise Modersohn, Martin Schoenthaler, Maximilian Glienke et al.
BMC Medical Informatics and Decision Making
Biomedical Text Mining and Ontologies
article

The RECUR corpus: annotated German radiology reports for urolithiasis research

Tabea Walther, Luise Modersohn, Martin Schoenthaler, Maximilian Glienke, Maximilian Ferry von Bargen, Lidija Ujkic, Raphael Schmitt
article en

Abstract

Background Advances in clinical NLP depend on high-quality datasets that reflect the complexity of real-world documentation. In response, a fine-grained, relation-annotated corpus for urinary stone diagnostics is introduced. The corpus addresses a gap in German medical NLP resources and provides a reference dataset for developing and evaluating clinical information extraction models. Methods We extracted and de-identified 262 radiology reports and 251 corresponding short findings concerning kidney and ureter stones from the hospital information system of the University Medical Center Freiburg. Using a detailed annotation schema comprising nine entity types and five relation types, three medical doctoral students annotated the corpus in Label Studio, followed by expert adjudication to establish a gold standard. For evaluation, we fine-tuned several German general-domain and medical language models for named entity recognition (NER) and relation extraction (RE) using a fixed train–development–test split, a unified Hugging Face pipeline, and NNI-based hyperparameter tuning. Results The resulting corpus comprises 513 radiology documents with 3,223 annotated entities, dominated by stone- and anatomy-related categories. Inter-annotator agreement was consistently high for both entity and relation annotations (NER F1 = 82.8%; RE F1 = 81.8%), supporting the reliability of the annotation schema. The evaluated models achieved macro F1-scores of up to 89.45% for NER and 99.24% for RE. While NER performance varied across models and fine-grained entity categories, RE performance was consistently high across all evaluated encoders. Conclusion RECUR establishes a fine-grained reference corpus and evaluation setting for automated information extraction from German radiology reports. Its adjudicated annotations, strong inter-annotator agreement, and comprehensive encoder baselines support the systematic evaluation of clinical NER and RE methods.

BMC Medical Informatics and Decision Making
Universidade dos Açores (PT), University of Freiburg (DE), University Medical Center Freiburg (DE), TUM Klinikum (DE), Technical University of Munich (DE)
Biomedical Text Mining and Ontologies
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon