Transferring the Knowledge of Vision-Language Model for Pathological Image Classification

Pathological image analysis is crucial for diagnosing a range of diseases, but it remains a complex and labor-intensive task requiring significant expertise. Traditional deep learning models for image analysis often rely on large, annotated datasets, which are costly to acquire and limit their applicability, especially when dealing with intricate morphological structures. Recently, large-scale pre-trained vision-language models (VLMs) have advanced multi-modal understanding by effectively integrating textual and visual information, enhancing the comprehension of complex visual data. Inspired by this, we propose PathCLS, a novel pathological image classification model that incorporates a fixed pre-trained VLM, PLIP, which is pre-trained on large-scale text-image pairs. PathCLS leverages domain-specific medical text knowledge by driving PLIP with text descriptions of images and further refining predictions with an MLP layer to bridge the domain gap between pre-training and downstream tasks. Our extensive experiments demonstrate that PathCLS significantly outperforms state-of-the-art methods, validating its effectiveness in enhancing pathological image classification.