MultI-MVQA: A multitask framework for interpretable medical visual question answering

Medical Visual Question Answering (MVQA) is a crucial task at the intersection of multimodal machine learning and the medical domain. An MVQA system aims to obtain an answer from an input question and image and can have many applications, from facilitating healthcare to advancing education. Given the nature of the medical field, the development of MVQA models presents specific challenges, including data limitations and the need for explainable results. As a step towards addressing these problems, we propose MultI-MVQA, a multitask framework designed to enhance the performance of an MVQA system across various base models. Our proposal integrates additional annotated data, such as question, answer, and organ types, within a cascaded multitask architecture. By applying our MultI-MVQA framework to two base models from the MVQA literature, we achieve improved performance. Furthermore, to enhance model interpretability, we use the Local Interpretable Model-agnostic Explanations algorithm for both textual and visual inputs, and present an analysis of the explanations from both computational and biological perspectives.