Adaptive Hard Negative Mining for Fine Grained Vision Language Understanding A Negative Result and Analysis

Abstract Vision-language models perform well on many multimodal tasks but struggle with fine-grained visual-linguistic reasoning. The SPEC benchmark was introduced to evaluate models on four fine-grained concepts: object size, position, existence, and count. Fine-tuning on SPEC with fixed hard negatives improves baseline performance substantially. This paper asks whether adaptive hard-negative mining, which reweights negatives by their current difficulty, can push performance further. We propose a confusion-weighted training objective and evaluate it on SPEC held-out data. The method underperforms fixed hard-negative training by 23 percentage points on average. We trace the cause to SPEC's design: its candidate sets are uniformly hard, so adaptive reweighting has no difficulty signal to exploit and instead destabilizes training. We also introduce SPEC++, a paired-reasoning diagnostic that combines cases from two SPEC subsets. The fine-tuned model's joint accuracy on SPEC + + equals the product of its individual accuracies, indicating that it treats concepts independently rather than composing them. These two findings serve as cautionary notes for the field: uniform hard negatives are difficult to outperform when the data is designed to be uniformly hard, and high benchmark accuracy should not be read as evidence of compositional understanding.