Revisiting LLMs and Lung Cancer Questions: How AI Responds to Common Lung Cancer Questions Two Years Later
GPT-o3-mini showed the best overall accuracy among the models tested but had notable inter-rater variability. While GPT-o3-mini and Gemini showed improved accuracy (75.83% and 74.17%) over their older iterations in prior assessments (GPT-3.5 70.8%, Bard 51.7%), ongoing issues persist. Nearly two years later, problems such as inconsistent performance, incorrect responses, and refusals emphasize the continued importance of expert oversight.
Authors
- Cameron Hassani (ORCID: https://orcid.org/0000-0002-2831-5384)
- Amir Ali Rahsepar (ORCID: https://orcid.org/0000-0002-6228-4673)
- Grace Hyun J. Kim (ORCID: https://orcid.org/0000-0003-1225-3489)
- Hamid Chalian (ORCID: https://orcid.org/0000-0001-8357-7347)
- Dan Nguyen (ORCID: https://orcid.org/0000-0002-9590-0655)
- Arash Bedayat (ORCID: https://orcid.org/0000-0001-8769-0224)
Institutions
- University of California, Los Angeles (US)
- University of Massachusetts Chan Medical School (US)
- University of Washington (US)
- Northwestern Memorial Hospital (US)
Publication Details
- Journal
- Academic Radiology
- Published
- 2025-09-16
- DOI
- https://doi.org/10.1016/j.acra.2025.08.056
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 0.00