From CNN to Vision Foundation Models and LLMs: A Multimodal Framework for Pathology Image Retrieval and Auto-Captioning
Background: Pathology image retrieval is crucial for diagnostic support, yet current vision-only retrieval systems often lack semantic interpretability and may generate hallucinated or confabulated outputs, limiting their clinical reliability. Captioning models like MedGemma produce textual descriptions but frequently diverge from expert annotations. To address these issues, we developed a multimodal framework that integrates vision foundation models with large language models (LLMs), leveraging retrieval-driven summarization.
Authors
- Hamid R. Tizhoosh (ORCID: https://orcid.org/0000-0001-5488-601X)
- Saghir Alfasly (ORCID: https://orcid.org/0000-0002-8561-660X)
- Arno Schmitgen
- Sahar Rahimi Malakshan (ORCID: https://orcid.org/0000-0002-6211-6039)
- Fred Prior (ORCID: https://orcid.org/0000-0002-6314-5683)
- Wataru Uegami (ORCID: https://orcid.org/0000-0002-0285-6270)
- Ghazal Alabtah
- Alireza Kazemi (ORCID: https://orcid.org/0000-0002-6194-7041)
- M. Fazlul Hoq
Institutions
- West Virginia University (US)
- University of Wisconsin–Madison (US)
- Princeton University (US)
- Mayo Clinic in Florida (US)
- Cellulose Sciences International (United States) (US)
Publication Details
- Journal
- Mayo Clinic Proceedings Digital Health
- Published
- 2025-12-01
- DOI
- https://doi.org/10.1016/j.mcpdig.2025.100291
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- article
- Field-Weighted Citation Impact
- 0.00