CLIP-ViT Detector: Side Adapter with Prompt for Vision Transformer Object Detection

Object detection represents a fundamental and pivotal task within the domain of computer vision, which has attracted considerable interest in approaches that directly utilize Vision Transformer (ViT) to perform region-level recognition. However, despite the efforts of early pioneers in exploring vanilla ViT detectors, a significant performance disparity remains. Addressing this limitation, in this work, we propose an ViT-based detector to further enhance the ability of plain ViT in object detection by incorporating a frozen vision-language large model. Specifically, to boost the ViT detector with the frozen CLIP model, we construct ViT-based Side Prompt-Adapter Tuning, which align and fine-tune CLIP features without requiring gradients to flow through CLIP to provide additional rich semantic information for the ViT detector. Furthermore, a CLIP-based visual token selection module is proposed to leverage fine-tuned CLIP features to filter out irrelevant background visual tokens, resulting in decreased computational complexity and memory usage of the ViT-based detector. Additionally, we introduce query denoising training and adapt the position embeddings to further enhance training efficiency. Compared to the latest ViT-based detector, experimental results show that our method converges 3× faster and achieves promising performance.