Efficient multimodal learning using BERT and vision transformers for visual question answering on peripheral blood cells

Peripheral blood cells are routinely examined in clinical practice, and answering clinically relevant questions about these cells is essential for decision support. In this work, we present an efficient multimodal learning framework for visual question answering on peripheral blood cells, combining BERT for textual representation with vision transformers for visual feature extraction. The proposed dual-stream architecture enables effective fusion of linguistic and visual information tailored to hematology data. To support this task, we construct a dedicated dataset by enriching an existing peripheral blood cell image collection with expert-annotated question–answer pairs. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art multimodal models, including BLIP and ViLT, achieving a WUPS score of 0.95, an F1 score of 0.94, and an accuracy of 0.95. These results establish a new benchmark for visual question answering in hematology and highlight the potential of efficient transformer-based multimodal models to enhance automated blood cell analysis and clinical interpretation.