A retrieval-augmented local large language model for ICD-10-CM terminology mapping of clinical signs and symptoms

Accurately mapping clinical signs and symptoms (S&S) from free text to ICD-10-CM codes is essential for early cancer detection, phenotyping, and risk-prediction modeling. However, manual coding is labor-intensive, and automated models often generate invalid or hallucinated codes. This study evaluates whether grounding a large language model (LLM) in the full ICD-10-CM catalog through retrieval-augmented generation (RAG) improves accuracy and reliability compared with an LLM operating without retrieval. Using 200 clinical notes containing 545 human-labeled S&S, we compared an LLM-only baseline, a retrieval-only voting baseline, and a RAG-LLM model. Eight domain-specific and general embedding models produced dense vectors for all 98,186 ICD-10-CM codes. For each symptom, up to 40 candidate codes were retrieved by cosine similarity and incorporated into the model prompt for final selection. Performance was evaluated at the three-character ICD-10 block level against human annotations, with expert adjudication of non-matching cases. Code validity and raw description consistency were evaluated using the official ICD-10-CM reference. The RAG-LLM model achieved 85.87% three-character ICD-10-CM match accuracy, outperforming the LLM-only baseline (72.11%) and the retrieval-only voting baseline (55.60%). After exploratory adjudication, 98.90% of RAG-LLM outputs were considered clinically acceptable S&S mappings. Paired McNemar tests showed significant improvements over both baselines. The RAG-LLM model also reduced invalid ICD-10-CM code generation from 9.91% in the LLM-only baseline to 1.10%. This proof-of-concept study shows that retrieval-guided LLM candidate selection can improve symptom/sign-level ICD-10-CM terminology mapping on MTsamples. The RAG-LLM model improved both three-character and full-code matching while reducing invalid ICD-10-CM code generation. These findings support further evaluation of retrieval-guided LLM mapping as a preprocessing approach for symptom-based phenotyping research, but additional validation is needed before use in real-world EHR workflows, billing-grade coding, or downstream disease-prediction models.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-20
DOI
https://doi.org/10.1038/s41598-026-74513-4
Primary Topic
Biomedical Text Mining and Ontologies
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A retrieval-augmented local large language model for ICD-10-CM terminology mapping of clinical signs and symptoms

Joseph Finkelstein, Yunbing Bai
4 links
Scientific Reports
Biomedical Text Mining and Ontologies
article

A retrieval-augmented local large language model for ICD-10-CM terminology mapping of clinical signs and symptoms

Joseph Finkelstein, Yunbing Bai
article en

Abstract

Accurately mapping clinical signs and symptoms (S&S) from free text to ICD-10-CM codes is essential for early cancer detection, phenotyping, and risk-prediction modeling. However, manual coding is labor-intensive, and automated models often generate invalid or hallucinated codes. This study evaluates whether grounding a large language model (LLM) in the full ICD-10-CM catalog through retrieval-augmented generation (RAG) improves accuracy and reliability compared with an LLM operating without retrieval. Using 200 clinical notes containing 545 human-labeled S&S, we compared an LLM-only baseline, a retrieval-only voting baseline, and a RAG-LLM model. Eight domain-specific and general embedding models produced dense vectors for all 98,186 ICD-10-CM codes. For each symptom, up to 40 candidate codes were retrieved by cosine similarity and incorporated into the model prompt for final selection. Performance was evaluated at the three-character ICD-10 block level against human annotations, with expert adjudication of non-matching cases. Code validity and raw description consistency were evaluated using the official ICD-10-CM reference. The RAG-LLM model achieved 85.87% three-character ICD-10-CM match accuracy, outperforming the LLM-only baseline (72.11%) and the retrieval-only voting baseline (55.60%). After exploratory adjudication, 98.90% of RAG-LLM outputs were considered clinically acceptable S&S mappings. Paired McNemar tests showed significant improvements over both baselines. The RAG-LLM model also reduced invalid ICD-10-CM code generation from 9.91% in the LLM-only baseline to 1.10%. This proof-of-concept study shows that retrieval-guided LLM candidate selection can improve symptom/sign-level ICD-10-CM terminology mapping on MTsamples. The RAG-LLM model improved both three-character and full-code matching while reducing invalid ICD-10-CM code generation. These findings support further evaluation of retrieval-guided LLM mapping as a preprocessing approach for symptom-based phenotyping research, but additional validation is needed before use in real-world EHR workflows, billing-grade coding, or downstream disease-prediction models.

Scientific ReportsVol. 16(1)
University of Arizona (US)
National Institutes of Health
Good health and well-being
Biomedical Text Mining and Ontologies
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon