Large Vision-Language Models are Generalist Solvers For Pathology Tasks

Leveraging the powerful capabilities of large language models (LLMs), large vision-language models (LVLMs) can perform a wide variety of tasks based on input images and user instructions. However, existing pathology-focused LVLMs are limited to relatively simple tasks, such as image captioning, visual question answering, and generating brief pathology reports, which restricts their clinical applicability. To enhance the practicality of pathology LVLMs and explore their performance boundaries across pathology tasks, we curated a multi-task pathology instruction-following dataset that better aligns with clinical needs. This dataset encompasses tasks such as cancer classification and grading, molecular subtype identification, and the detection of structures like nuclei, blood vessels, nerves, and lymph nodes. Extensive experiments were conducted on this dataset to identify key factors influencing the performance of LVLMs on these pathology tasks, and optimal solutions were proposed. Our findings provide valuable insights to advance the clinical application of large vision-language models in pathology.