A retrieval-augmented local large language model for ICD-10-CM terminology mapping of clinical signs and symptoms
Accurately mapping clinical signs and symptoms (S&S) from free text to ICD-10-CM codes is essential for early cancer detection, phenotyping, and risk-prediction modeling. However, manual coding is labor-intensive, and automated models often generate invalid or hallucinated codes. This study evaluates whether grounding a large language model (LLM) in the full ICD-10-CM catalog through retrieval-augmented generation (RAG) improves accuracy and reliability compared with an LLM operating without retrieval. Using 200 clinical notes containing 545 human-labeled S&S, we compared an LLM-only baseline, a retrieval-only voting baseline, and a RAG-LLM model. Eight domain-specific and general embedding models produced dense vectors for all 98,186 ICD-10-CM codes. For each symptom, up to 40 candidate codes were retrieved by cosine similarity and incorporated into the model prompt for final selection. Performance was evaluated at the three-character ICD-10 block level against human annotations, with expert adjudication of non-matching cases. Code validity and raw description consistency were evaluated using the official ICD-10-CM reference. The RAG-LLM model achieved 85.87% three-character ICD-10-CM match accuracy, outperforming the LLM-only baseline (72.11%) and the retrieval-only voting baseline (55.60%). After exploratory adjudication, 98.90% of RAG-LLM outputs were considered clinically acceptable S&S mappings. Paired McNemar tests showed significant improvements over both baselines. The RAG-LLM model also reduced invalid ICD-10-CM code generation from 9.91% in the LLM-only baseline to 1.10%. This proof-of-concept study shows that retrieval-guided LLM candidate selection can improve symptom/sign-level ICD-10-CM terminology mapping on MTsamples. The RAG-LLM model improved both three-character and full-code matching while reducing invalid ICD-10-CM code generation. These findings support further evaluation of retrieval-guided LLM mapping as a preprocessing approach for symptom-based phenotyping research, but additional validation is needed before use in real-world EHR workflows, billing-grade coding, or downstream disease-prediction models.
Authors
- Joseph Finkelstein (ORCID: https://orcid.org/0000-0002-8084-7441)
- Yunbing Bai (ORCID: https://orcid.org/0009-0005-4910-4497)
Institutions
- University of Arizona (US)
Publication Details
- Journal
- Scientific Reports
- Published
- 2026-11-20
- DOI
- https://doi.org/10.1038/s41598-026-74513-4
- Primary Topic
- Biomedical Text Mining and Ontologies
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- National Institutes of Health