Red teaming ChatGPT in medicine to yield real-world insights on model behavior
Red teaming, the practice of adversarially exposing unexpected or undesired model behaviors, is critical towards improving equity and accuracy of large language models, but non-model creator-affiliated red teaming is scant in healthcare. We convened teams of clinicians, medical and engineering students, and technical professionals (80 participants total) to stress-test models with real-world clinical cases and categorize inappropriate responses along axes of safety, privacy, hallucinations/accuracy, and bias. Six medically-trained reviewers re-analyzed prompt-response pairs and added qualitative annotations. Of 376 unique prompts (1504 responses), 20.1% were inappropriate (GPT-3.5: 25.8%; GPT-4.0: 16%; GPT-4.0 with Internet: 17.8%). Subsequently, we show the utility of our benchmark by testing GPT-4o, a model released after our event (20.4% inappropriate). 21.5% of responses appropriate with GPT-3.5 were inappropriate in updated models. We share insights for constructing red teaming prompts, and present our benchmark for iterative model assessments.
Authors
- Benjamin H. Jacobson (ORCID: https://orcid.org/0000-0002-5550-3559)
- Akash Chaurasia (ORCID: https://orcid.org/0000-0003-3438-3313)
- Solveig Behr
- Jesutofunmi A. Omiye (ORCID: https://orcid.org/0000-0002-0659-831X)
- Jacob Chang
- Luca Pegolotti (ORCID: https://orcid.org/0000-0002-4290-6391)
- Irena Gao
- Charbel Bou-Khalil (ORCID: https://orcid.org/0000-0002-5497-9037)
- Juan M. Banda (ORCID: https://orcid.org/0000-0001-8499-824X)
- Archer Y. Yang
- Matthew Schwede (ORCID: https://orcid.org/0000-0002-0778-9230)
- Roxana Daneshjou (ORCID: https://orcid.org/0000-0001-7988-9356)
- Lü Tian (ORCID: https://orcid.org/0000-0002-5893-0169)
- Dhamanpreet Kaur (ORCID: https://orcid.org/0000-0001-6123-7499)
- Celeste H. Poe
- Balasubramanian Narasimhan (ORCID: https://orcid.org/0000-0001-5852-7639)
- Jason Fries (ORCID: https://orcid.org/0000-0001-9316-5768)
- Cheng Ge (ORCID: https://orcid.org/0000-0003-4010-5686)
- Alejandro Lozano (ORCID: https://orcid.org/0000-0002-6952-289X)
- Rika Bajra (ORCID: https://orcid.org/0000-0002-1378-2561)
- Alice Heiman
- Angela Y. Jia (ORCID: https://orcid.org/0000-0002-0812-2614)
- Nigam H. Shah (ORCID: https://orcid.org/0000-0001-9385-7158)
- Allison Sihan Jia
- Yuqi Tan (ORCID: https://orcid.org/0000-0002-7200-3222)
- Tiffany Eulalio (ORCID: https://orcid.org/0000-0002-7084-9646)
- Jonathan H. Chen (ORCID: https://orcid.org/0000-0002-4387-8740)
- Akaash Kolluri
- Eric Lin (ORCID: https://orcid.org/0000-0002-1061-7704)
- John Tamaresis (ORCID: https://orcid.org/0000-0002-9868-2366)
- Ye-Jean Park
- Fateme Nateghi Haredasht (ORCID: https://orcid.org/0000-0002-8874-8835)
- Sang Truong (ORCID: https://orcid.org/0000-0001-8069-9410)
- Shawheen J. Rezaei (ORCID: https://orcid.org/0000-0002-4394-7371)
- Chenming Zheng
- Yutong Liu (ORCID: https://orcid.org/0009-0003-6695-8991)
- Hodan Farah (ORCID: https://orcid.org/0009-0001-5150-2153)
- Haiwen Gui (ORCID: https://orcid.org/0000-0003-0564-940X)
- Minghe Zhang
- Yanan Ding (ORCID: https://orcid.org/0000-0001-6025-6575)
- Chandan Sanghera (ORCID: https://orcid.org/0000-0002-7060-442X)
- Akshay Swaminathan (ORCID: https://orcid.org/0000-0003-3426-9289)
- Diana Hla (ORCID: https://orcid.org/0000-0002-9562-294X)
- J. Weston Jamison
- Malathi Srinivasan (ORCID: https://orcid.org/0000-0001-9951-0016)
- Binglan Li (ORCID: https://orcid.org/0000-0002-0103-6107)
- Scott L. Fleming (ORCID: https://orcid.org/0000-0002-6047-7877)
- Kirsten R. Steffner (ORCID: https://orcid.org/0000-0001-9759-616X)
- Mohsen Fathzadeh (ORCID: https://orcid.org/0000-0002-7962-9148)
- Amit Kaushal (ORCID: https://orcid.org/0000-0003-0831-4365)
- Maniragav Manimaran
- Angelica Iacovelli (ORCID: https://orcid.org/0009-0004-1327-2184)
- Benjamin Belai
- Damien Varid Kettud
- Diba Dindoust
- Dominik Moehrle
- Joshua Lazaro
- Karen Ebert Matthys
- Ruhana Azam
- Sonia Onyeka
- Tarek Bandali
- Ying Cui (ORCID: https://orcid.org/0000-0002-3697-5155)
- Xuanyu Zhou
- Minh Hoai Nguyen (ORCID: https://orcid.org/0000-0001-7149-849X)
- Crystal Chang (ORCID: https://orcid.org/0000-0002-0647-7040)
- Julie Lee (ORCID: https://orcid.org/0009-0002-6640-8597)
- Sri Muppidi
- Jiapeng Xu (ORCID: https://orcid.org/0000-0002-0970-7606)
- Arpita Singhal (ORCID: https://orcid.org/0000-0002-6515-6474)
- Rohit Khurana
- Ellaine Chou
- Deven Pandya
- R. W. Bartlett
- Arghavan Salles
- Ross Duncan (ORCID: https://orcid.org/0009-0003-7502-1279)
- Conor Messer
- Jake Silberg
- S. Varadha Raj
- Wenyuan Chen
- Qian Zhao (ORCID: https://orcid.org/0000-0002-5179-7836)
Institutions
- Mayo Clinic (US)
- Harvard University (US)
- Palo Alto University (US)
- Veterans Health Administration (US)
- Stanford Health Care (US)
- Mayo Clinic in Arizona (US)
- Stanford Medicine (US)
- Mayo Clinic in Florida (US)
- McGill University (CA)
- Freie Universität Berlin (DE)
- Stanford University (US)
Publication Details
- Journal
- npj Digital Medicine
- Published
- 2025-03-07
- DOI
- https://doi.org/10.1038/s41746-025-01542-0
- Citations
- 23
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 15.05