Memorization in large language models in medicine prevalence characteristics and implications

Large Language Models (LLMs) have demonstrated significant potential in medicine, with many studies adapting them through continued pretraining or fine-tuning on medical data. However, a key question remains: to what extent do LLMs memorize medical training data-that is, recall or regenerate content seen during continued pretraining or fine-tuning. In this work, we investigate memorization of LLMs in medicine, assessing its prevalence (frequency), characteristics (what is memorized), volume (how much), and potential downstream impacts. We systematically analyze common adaptation scenarios: (1) continued pretraining on medical corpora, (2) fine-tuning on standard medical benchmarks, and (3) fine-tuning on real-world clinical data, including over 13,000 unique inpatient records from Yale New Haven Health System. The results demonstrate that memorization is prevalent and significantly higher than that in the general domain. Memorization has distinct characteristics during continued pretraining and fine-tuning, and it is persistent: up to 87% of content memorized during continued pretraining remains after fine-tuning. Memorization can be categorized into three types: beneficial (e.g., accurate recall of clinical guidelines), uninformative (e.g., templated language), and harmful (e.g., sensitive clinical content). We offer practical recommendations to facilitate beneficial memorization, minimize uninformative memorization, and mitigate harmful memorization to protect patient privacy and improve medical utility.

Authors

Institutions

Publication Details

Journal
Nature Communications
Published
2026-06-19
DOI
https://doi.org/10.1038/s41467-026-73779-6
Citations
1
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Field-Weighted Citation Impact
6.59

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Memorization in large language models in medicine prevalence characteristics and implications

Lingfei Qian, Anran Li, Jimin Huang, Yan Hu et al.
1 citations
Nature Communications
Artificial Intelligence in Healthcare and Education
6.59
preprint

Memorization in large language models in medicine prevalence characteristics and implications

Lingfei Qian, Anran Li, Jimin Huang, Yan Hu, Zhong‐Lin Lu, Qingyu Chen, Yangyang Fu, Lucila Ohno‐Machado, Yih Chung Tham, Meirong Du, Siru Liu, Yin Yu, Hua Xu, Qianqian Xie, Rui Zhu, Zihao Sun, Yifan Yang, X. C. Ai, H.S. Cho, Erica Stutz
preprint en
1 citations

Abstract

Large Language Models (LLMs) have demonstrated significant potential in medicine, with many studies adapting them through continued pretraining or fine-tuning on medical data. However, a key question remains: to what extent do LLMs memorize medical training data-that is, recall or regenerate content seen during continued pretraining or fine-tuning. In this work, we investigate memorization of LLMs in medicine, assessing its prevalence (frequency), characteristics (what is memorized), volume (how much), and potential downstream impacts. We systematically analyze common adaptation scenarios: (1) continued pretraining on medical corpora, (2) fine-tuning on standard medical benchmarks, and (3) fine-tuning on real-world clinical data, including over 13,000 unique inpatient records from Yale New Haven Health System. The results demonstrate that memorization is prevalent and significantly higher than that in the general domain. Memorization has distinct characteristics during continued pretraining and fine-tuning, and it is persistent: up to 87% of content memorized during continued pretraining remains after fine-tuning. Memorization can be categorized into three types: beneficial (e.g., accurate recall of clinical guidelines), uninformative (e.g., templated language), and harmful (e.g., sensitive clinical content). We offer practical recommendations to facilitate beneficial memorization, minimize uninformative memorization, and mitigate harmful memorization to protect patient privacy and improve medical utility.

Nature Communications
National Institutes of Health (US), National University of Singapore (SG), Vanderbilt University (US), United States National Library of Medicine (US), Yale University (US), University of California San Diego (US), Yale New Haven Health System (US), The University of Texas Health Science Center (US), Imperial College London (GB), Vanderbilt University Medical Center (US), The University of Texas Health Science Center at Houston (US)
Foundation for the National Institutes of Health, National Institutes of Health, U.S. National Library of Medicine
Openalex Percentile: Top 8%
Artificial Intelligence in Healthcare and Education
6.59
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.