Optimizing Transformer and MLP with Hidden States Perturbation for Medical Visual Question Answering

Optimizing model performance is a crucial objective in medical visual question answering (MedVQA), and a wide range of techniques have been developed to achieve this goal. In this paper, we propose a novel technique called network state perturbation, which distinguishes itself from existing research. Our study designs four innovative methods to modify the hidden states within the network and improve the performance of multimodal models in the MedVQA task. Specifically, we evaluate these methods on both general transformer and multilayer perceptron (MLP) models, which allow for hidden state adjustments at each layer. The four introduced methods are as follows: (1) Randomly Set Zero, which assigns zeros to the hidden states of different modalities in the network; (2) Randomly Replace Content, which performs interpolation between the hidden states of the text sequence and the image sequence; (3) Randomly Add Gaussian Noise, which adds Gaussian noise to the hidden states of different modalities; and (4) Pair Interpolation, which interpolates the hidden states of different modalities of the current sample with those of other samples and performs corresponding label interpolation to facilitate model training. Our experimental results demonstrate that the proposed hidden state perturbation methods significantly enhance the performance of various transformer and MLP models on multiple MedVQA datasets, without requiring additional data or computational resources during training. These findings highlight the potential of hidden state perturbation as a novel model improvement technique for multimodal models in the medical domain.