Collaboration between clinicians and vision–language models in radiology report generation
Automated radiology report generation has the potential to improve patient care and reduce the workload of radiologists. However, the path toward real-world adoption has been stymied by the challenge of evaluating the clinical quality of artificial intelligence (AI)-generated reports. We build a state-of-the-art report generation system for chest radiographs, called Flamingo-CXR, and perform an expert evaluation of AI-generated reports by engaging a panel of board-certified radiologists. We observe a wide distribution of preferences across the panel and across clinical settings, with 56.1% of Flamingo-CXR intensive care reports evaluated to be preferable or equivalent to clinician reports, by half or more of the panel, rising to 77.7% for in/outpatient X-rays overall and to 94% for the subset of cases with no pertinent abnormal findings. Errors were observed in human-written reports and Flamingo-CXR reports, with 24.8% of in/outpatient cases containing clinically significant errors in both report types, 22.8% in Flamingo-CXR reports only and 14.0% in human reports only. For reports that contain errors we develop an assistive setting, a demonstration of clinician-AI collaboration for radiology report composition, indicating new possibilities for potential clinical utility.
Authors
- Shravya Shetty (ORCID: https://orcid.org/0000-0003-3783-3172)
- Danielle Belgrave
- Pushmeet Kohli (ORCID: https://orcid.org/0000-0002-7466-7997)
- Tao Tu (ORCID: https://orcid.org/0000-0001-9191-7938)
- Mike Schaekermann (ORCID: https://orcid.org/0000-0002-1735-9680)
- Andrew Sellergren (ORCID: https://orcid.org/0000-0002-5260-8013)
- S. M. Ali Eslami (ORCID: https://orcid.org/0000-0003-1838-5589)
- Sreenivasa Raju Kalidindi (ORCID: https://orcid.org/0000-0001-5787-1956)
- Joelle K. Barral (ORCID: https://orcid.org/0009-0009-0432-5148)
- Shekoofeh Azizi (ORCID: https://orcid.org/0000-0002-7447-6031)
- Sumanth Dathathri (ORCID: https://orcid.org/0009-0007-4937-9903)
- Charles T. Lau (ORCID: https://orcid.org/0000-0002-3136-9711)
- S. Sara Mahdavi (ORCID: https://orcid.org/0000-0001-6823-598X)
- Ryutaro Tanno (ORCID: https://orcid.org/0000-0002-8107-6730)
- Yossi Matias (ORCID: https://orcid.org/0000-0003-3960-6002)
- Johannes Welbl
- Sumedh Ghaisas (ORCID: https://orcid.org/0000-0003-3753-9029)
- Sofia Ira Ktena (ORCID: https://orcid.org/0000-0001-6677-6547)
- Po-Sen Huang (ORCID: https://orcid.org/0000-0003-1470-0991)
- Roy Lee (ORCID: https://orcid.org/0000-0001-7602-3979)
- Abigail See (ORCID: https://orcid.org/0000-0003-3137-6599)
- Yun Liu (ORCID: https://orcid.org/0000-0003-4079-8275)
- K. K. Singhal (ORCID: https://orcid.org/0009-0001-0286-609X)
- Alan Karthikesalingam (ORCID: https://orcid.org/0009-0000-4958-5976)
- Zahra Saeed Ahmed (ORCID: https://orcid.org/0000-0003-3712-5125)
- Vivek Natarajan (ORCID: https://orcid.org/0000-0001-7849-2074)
- SiWai Man
- David G. T. Barrett
- R. May
Institutions
- GlaxoSmithKline (United Kingdom) (GB)
- Google DeepMind (United Kingdom) (GB)
- Google (United Kingdom) (GB)
- OpenAI (United States) (US)
Publication Details
- Journal
- Nature Medicine
- Published
- 2024-11-07
- DOI
- https://doi.org/10.1038/s41591-024-03302-1
- Citations
- 116
- Primary Topic
- Radiology practices and education
- Type
- article
- Field-Weighted Citation Impact
- 46.90
Funders
- DeepMind