DTC: Difference-aware Transformer with CLIP Adaptation for Change Captioning

Existing Change Captioning (CC) methods typically rely on frozen pre-trained vision foundation models to extract visual difference features and utilize the traditional transformer's global attention mechanism to establish cross-modal mappings between visual differences and change captions. Although these methods have achieved significant success, they still face the following challenges: (1) Vision foundation models are typically pre-trained on single images, making it difficult to perceive and encode changes between two similar images without fine-tuning; (2) There are numerous redundant regions irrelevant to changes leading to difficulties in accurately locating key regions of interest for changes using traditional global attention while also incurring high computational costs. To address these challenges, we propose a novel Difference-aware Transformer with CLIP Adaptation (DTC) for the CC task. Technically, DTC first designs a mutual retrieval loss between visual differences and change captions to fine-tune the CLIP model, making it better suited to the characteristics of CC and the diversity of datasets. This cross-modal adaptation allows our model to learn the visual difference features between two images more effectively. Additionally, DTC proposes a difference-aware attention mechanism that extends the global attention's triplet computation into a quadriplet paradigm to enhance difference modeling. Specifically, by introducing difference-aware tokens, which are far fewer than the full set of queries, key information is first aggregated from the key and value features to filter out redundant information. This key information is then propagated to the full query set along with the difference-aware tokens as new value features, enhancing the spread of critical change features and improving the accuracy of generated captions. Notably, compared to global attention, the proposed difference-aware attention significantly reduces computational complexity (from O(N²d) to O(NMd), where M<<N). Extensive experimental results on multiple datasets demonstrate that DTC outperforms state-of-the-art (SOTA) methods in both performance and computational efficiency, proving its effectiveness and potential in the CC task.