Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA

Random GuessPosition w/o adversarial pair w/ adversarial pair Figure 1: Accuracy of six LMMs on two types of specialized questions in medical diagnoses, with and without adversarial pairs.The significant drop in accuracy with adversarial pairs highlights the models' unreliability in handling medical diagnoses with our probing method.