Medical visual question answering based on Transformer and Mamba fusion Architecture

Medical Visual Question Answering (Med-VQA) aims to analyze medical images (such as CT, MRI, X-ray) and answer relevant questions raised by users. The introduction of Transformer, deep learning has undergone tremendous changes, and it has achieved remarkable success in many fields such as natural language processing and computer vision. And it has been more commonly used in medical image analysis. Although Transformer and its variant models have been proven to be effective in many fields, the traditional Transformer architecture has a performance bottleneck in long sequence context reasoning due to the computational complexity of its attention mechanism, which is proportional to the square of the sequence length N2, limiting its scalability. On the other hand, when the same patient asks multiple questions, the questions can be viewed as a long sequence, and understanding multiple questions to give more accurate answers is necessary in Med-VQA. The Transformer-based method sometimes may not be able to give an accurate answer. Recently, Mamba, as the latest state-space model SSM, provides a better solution. However, the limitation of Mamba is that it focuses on temporal modeling, is not naturally adapted to complex multimodal interaction tasks, and is generally not as expressive as Transformer for high-dimensional data. To this end, this paper proposes a visual question answering model based on the multi-modal and multi-scale Transformer-Mamba architecture called MM-Tramba. To address the insufficient utilization of visual information in the existing Med-VQA framework, this model uses multi-scale visual feature extraction and extracts unstructured features through the SS2D module to obtain richer visual information. It then uses the Transformer and Mamba fusion architecture to fuse modal information to achieve better performance. This paper conducts multiple experiments on the benchmark datasets VQA-RAD [1] and SLAKE-EN [2]. Experimental results show that MM-Tramba is not only a powerful model with excellent performance on long sequence problems, but also helps to greatly expand the Transformer and Mamba fusion architecture in the multimodal field.