Understanding the robustness of vision-language models to medical image artefacts
Vision-language models (VLMs) show promise for answering clinically relevant questions, but their robustness to medical image artefacts remains unclear. We evaluated VLMs' robustness through their performance on images with and without weak artefacts across five artefact categories, as well as their ability to detect images with strong artefacts. We built evaluation benchmarks using brain MRI scans, Chest X-ray, and retinal images, involving four real-world medical datasets. VLMs achieved moderate accuracy on original unaltered images (0.645, 0.602 and 0.604 for MRI, OCT, and X-ray applications, respectively). Accuracy declined with weak artefacts (-3.34%, -9.06% and -10.46%), while strong artefacts were detected at low rates (0.194, 0.128 and 0.115). Our findings indicated that VLMs are not yet capable of performing tasks on medical images with artefacts, underscoring the need to establish uniform benchmark thoroughly examining model robustness to image artefacts, and explicitly incorporate artefact-aware method design and robustness tests into VLM development.
Authors
- Pearse A. Keane (ORCID: https://orcid.org/0000-0002-9239-745X)
- Jocelyn Hui Lin Goh (ORCID: https://orcid.org/0000-0002-5052-6081)
- Hanyuan Zhang (ORCID: https://orcid.org/0000-0002-1527-9468)
- An Ran Ran (ORCID: https://orcid.org/0000-0003-4592-4867)
- Carol Y. Cheung (ORCID: https://orcid.org/0000-0002-9672-1819)
- Siegfried K. Wagner (ORCID: https://orcid.org/0000-0003-4915-4353)
- Daniel C. Alexander (ORCID: https://orcid.org/0000-0003-2439-350X)
- Zijie Cheng
- David A. Merle (ORCID: https://orcid.org/0000-0003-4485-5580)
- Ariel Yuhan Ong (ORCID: https://orcid.org/0000-0001-9300-573X)
- Ke Zou (ORCID: https://orcid.org/0000-0002-1181-1779)
- Liren Pang
- Hongyang Jiang (ORCID: https://orcid.org/0000-0003-0287-8581)
- André Altmann (ORCID: https://orcid.org/0000-0002-9265-2393)
- Boxuan Li (ORCID: https://orcid.org/0000-0001-9510-1098)
- Sahana Srinivasan (ORCID: https://orcid.org/0009-0004-9355-6717)
- Yih Chung Tham (ORCID: https://orcid.org/0000-0002-6752-797X)
- Yukun Zhou (ORCID: https://orcid.org/0000-0002-0840-6422)
- Lie Ju
- Robert Chen (ORCID: https://orcid.org/0000-0002-8371-8629)
- Tiantian He (ORCID: https://orcid.org/0000-0002-1604-3131)
- Dawei Yang (ORCID: https://orcid.org/0009-0002-0619-5059)
Institutions
- Moorfields Eye Hospital NHS Foundation Trust (GB)
- National University of Singapore (SG)
- Chinese University of Hong Kong (HK)
- Beijing Jiaotong University (CN)
- Singapore National Eye Center (SG)
- Singapore Eye Research Institute (SG)
- Duke-NUS Medical School (SG)
- UK Dementia Research Institute (GB)
- The London College (GB)
- Moorfields Eye Hospital (GB)
- University College London (GB)
Publication Details
- Journal
- npj Digital Medicine
- Published
- 2025-11-27
- DOI
- https://doi.org/10.1038/s41746-025-02108-w
- Citations
- 3
- Primary Topic
- Domain Adaptation and Few-Shot Learning
- Type
- article
- Field-Weighted Citation Impact
- 7.44
Funders
- Wellcome Trust
- Moorfields Eye Charity