Using Text-augmented Visual Prompt Learning for Histopathology Image Classification

In order to fully utilize the visual language to pretrain the model, while avoiding the use of a large dataset to re-train the model. When using visual prompt learning to classify histopathology images, pure visual prompts lack multi-perspective semantic information and have domain gaps. This leads to the proposed TeV-PLM approach. Text-enhanced visual prompts are used during image encoding. Textual information is first incorporated into the visual prompts and then inserted together into the image embedding sequence. A filtering module is added after image encoding to remove redundant information from the image embedding and retain important features. The textual information in the prompt comes from the text encoding branch. And the symptom descriptions of the categories are additionally added to the text encoding to more accurately distinguish between different categories. Finally, multiple experiments are conducted on the PatchGastrics gastric adenocarcinoma dataset, and compared with several benchmark models, the experimental results show a significant improvement in model performance, with an average accuracy of up to 54.98%, which is an improvement of 3.61% compared to the CITE model.