MMKNet: A Multi-Modal Knowledge Network for Predicting Both Seen and Unseen Classes in Medical Imaging

Generalized zero-shot learning (ML-GZSL) has demonstrated significant potential in medical diagnostics due to doctors' need to process large volumes of medical images. Vision Transformers (ViTs), due to their Transformer-like structure, are considered to have superior feature-generation capabilities in cross-text-image tasks. BioMedBERT, based on the BERT architecture and domain-specific pre-training for biomedical natural language processing, is considered to have significant label embedding capabilities in cross-text-image tasks. In this paper, we propose MMKNet, a novel method that employs ViTs to construct global and local features of images for visual knowledge from images while using BioMedBERT with prompt tuning for the label embedding to achieve the knowledge from textual embedding in biomedical corpora. To integrate multi-modal information, we design a unique combined decision layer, which outputs similarity scores between images and class labels, providing the predicted classifications. Our method is class-independent during inference, enabling the model to predict unseen classes. Experiments on the NIH-ChestXray14, Kaggle retina, and Multi-Label Retinal Diseases (MuReD) datasets demonstrate that our method outperforms baseline models across multiple performance metrics, which can potentially optimize doctors' workflows by allowing them to focus on diagnosing complex cases, addressing challenges of limited dataset sizes and incomplete disease coverage in the medical imaging domain.