Vulnerability of Large Language Models to Prompt Injection When Providing Medical Advice
In this quality improvement study using a controlled simulation, commercial LLMs demonstrated substantial vulnerability to prompt-injection attacks that could generate clinically dangerous recommendations; even flagship models with advanced safety mechanisms showed high susceptibility. These findings underscore the need for adversarial robustness testing, system-level safeguards, and regulatory oversight before clinical deployment.
Authors
- Jungyo Suh (ORCID: https://orcid.org/0000-0002-3867-4778)
- Jeong-Moo Lee
- Tae Joon Jun (ORCID: https://orcid.org/0000-0002-6808-5149)
- Ro Woon Lee
- Hyungjun Park (ORCID: https://orcid.org/0000-0003-4412-3830)
- Soo Ick Cho
Institutions
- Ulsan College (KR)
- Ewha Womans University (KR)
- Inha University (KR)
- Asan Medical Center (KR)
- University of Ulsan (KR)
- Hanby Environmental (United States) (US)
- Institute for National Security Strategy (KR)
- Artificial Intelligence in Medicine (Canada) (CA)
Publication Details
- Journal
- JAMA Network Open
- Published
- 2025-12-19
- DOI
- https://doi.org/10.1001/jamanetworkopen.2025.49963
- Citations
- 4
- Primary Topic
- Adversarial Robustness in Machine Learning
- Type
- article
- Field-Weighted Citation Impact
- 9.92