Optimizing medical visual question answering: Evaluating the impact of enhanced images, augmented training data, and model selection

Abstract Visual question answering ( VQA ) has an interesting application in clinical decision support and enables clinicians to extract information from medical images through natural language queries. However, the limited nature of the datasets makes it particularly difficult to develop effective VQA models for the medical profession. The aim of this study was to overcome these obstacles by formally testing methods for data enhancement and model optimization. Specifically, we merged two medical VQA datasets, applied image preprocessing techniques, examined several state‐of‐the‐art model architectures, and extensively trained the best‐performing model on the enhanced data. The results showed that training the VGG16 ‐ LSTM model on sharper images than the merged dataset resulted in a significant performance improvement due to extending the training time to 200, with F1 scores of the training set 0.9674.