The Enhanced vision-language model developing for fine-grained difficult sample classification tasks

In special practical scenarios such as severe weather and dense buildings, intelligent target recognition relying on single modal images faces problems of image occlusion and blurring. Especially for fine-grained targets, the classification accuracy will significantly decrease. Based on the complementary characteristics of multimodal information, an enhanced vision-language model for fine-grained target recognition is proposed (DS-VL+). The experimental results show that randomly deactivating modal information during the training process and using feature fusion and decision fusion methods for target recognition, can effectively address the problem of modal loss. This article also proposes a method for constructing the difficult sample dataset that simulates complex real-world data situations, and constructs two datasets: Difficulty-CUB and Difficulty-Flower. The influence of difficult samples on the accuracy of single mode classification algorithm is analyzed on these two data sets. The classification accuracy of this model on fine-grained difficult samples can be comparable to that of unimodal models on general samples, while surpassing existing image language classification models and achieving SOTA.