The effectiveness of large language models in medical AI research for physicians: A randomized controlled trial
Physicians offer invaluable clinical insights, but their involvement in medical AI research is hindered by limited technical expertise. We conduct a superiority, open-label, randomized controlled trial involving 64 junior ophthalmologists to undertake a 2-week project on "automated cataract identification" under minimal engineering assistance, with (intervention, n = 32) or without (control, n = 32) ChatGPT-3.5. The overall project completion rate is higher in intervention group than controls (87.5% vs. 25.0%; difference 62.5%, p = 9.42e-7), and the unassisted completion rate likewise (68.7% vs. 3.1%; difference 65.6%, p = 5.70e-8). The intervention group demonstrates better project planning and faster completion times (p < 0.01). After a 2-week washout, 41.2% of successful intervention participants complete a new project without the support of large language models (LLMs). A survey shows that 42.6% of participants fear regurgitating information without understanding and 40.4% worry about fostering lazy thinking, indicating potential dependency. Therefore, LLMs can help physicians overcome technical barriers, although long-term risks require further study. Trial registration: This study was registered at ClinicalTrials.gov (NCT06015178).
Authors
- Yunxi Lai (ORCID: https://orcid.org/0000-0002-5709-2351)
- Miaohong Xu
- Zizheng Cao (ORCID: https://orcid.org/0000-0002-4774-7545)
- Wenben Chen
- Yuanfan Lin (ORCID: https://orcid.org/0009-0005-2295-3827)
- Andi Xu (ORCID: https://orcid.org/0009-0007-8466-1074)
- Sijia Yu (ORCID: https://orcid.org/0000-0002-3937-6676)
- Jianyu Pang (ORCID: https://orcid.org/0009-0007-2483-6717)
- Haotian Lin (ORCID: https://orcid.org/0000-0003-4672-9721)
- Lanqin Zhao (ORCID: https://orcid.org/0000-0002-5182-3678)
- Yuanjun Shang
- Shaowei Bi (ORCID: https://orcid.org/0009-0002-9372-6021)
- Yuanrui Shang
- Yuchun He
- Dong Liu (ORCID: https://orcid.org/0000-0002-0619-1480)
- Mingyuan Li (ORCID: https://orcid.org/0000-0001-9641-9284)
- Ruiyang Li (ORCID: https://orcid.org/0000-0003-4274-3841)
- Xinwei Chen (ORCID: https://orcid.org/0000-0001-5443-0215)
- Mingjie Luo (ORCID: https://orcid.org/0000-0001-7482-4975)
- Qianni Wu
- Jiaman Zhao
- Ling Jin
Institutions
- Fujian Medical University (CN)
- Sun Yat-sen University (CN)
- Hainan Eye Hospital (CN)
- Second Affiliated Hospital of Fujian Medical University (CN)
Publication Details
- Journal
- Cell Reports Medicine
- Published
- 2025-11-26
- DOI
- https://doi.org/10.1016/j.xcrm.2025.102469
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- National Supercomputer Centre in Guangzhou
- National Natural Science Foundation of China
- Sun Yat-sen University
- China Postdoctoral Science Foundation
- Basic and Applied Basic Research Foundation of Guangdong Province