Enhanced spatial awareness of vision-language model for car damage detection

Car damage detection aims to identify and locate various types of damage on the vehicle, such as dents, scratches, and cracks. Existing works tend to employ deep learning-based object detection approaches for car damage detection, i.e., Mask R-convolutional neural network. With the emergence of large-scale vision-language models (VLMs) such as contrastive language-image pretraining (CLIP), transferring these large-scale models to downstream tasks has become the most popular paradigm in the current field of deep learning. We aim to leverage pre-trained CLIP for car damage detection. However, directly using CLIP is prone to overfitting the training distribution, which is attributed to the absence of robust spatial awareness ability. To this end, we propose to enhance spatial awareness of CLIP from two aspects. First, we propose the spatial-aware feature injection (SFI) that pre-trains a car parts segmentation network to obtain spatial-aware features which are injected into different layers of CLIP’s image encoder through our spatial-aware low-rank cross-attention (SLCA) block. On the contrary, the latent image-text alignment characteristic in CLIP enables us to customize the input for the CLIP’s text encoder. Subsequently, we propose the cross-modal object-contextual representation (COR) module which consists of a cross-modal alignment module to facilitate the alignment of image and text features and an object-contextual refinement module to establish object-contextual representation by the aligned features. The experimental results consistently demonstrate the superiority of our approach compared with existing methods.