A Text-Guided Query Adaptive Vision Transformer for Pansharpening

Pansharpening aims to fuse a low resolution multispectral (LRMS) image and a high resolution panchromatic (PAN) image, and then synthesize a high resolution MS (HRMS) image. Benefiting from its potent capacity for global information modeling, vision Transformer (ViT) has achieved remarkable performance in pansharpening. However, most ViT-based methods heavily rely on cross-attention with static modality queries to extract and integrate complementary information from PAN and LRMS images, neglecting the adaptivity and specificity of different modalities. Moreover, vision-only representation framework often struggles to balance spatial enhancement with spectral fidelity. To tackle these issues, this letter proposes a Text guided Query Adaptive ViT (Text-QAViT) for pansharpening. Specifically, a Query Adaptive Cross-Attention (QACA) block is designed to enhance cross-modal interaction, which comprises cross-modal query projection and gating selection mechanism. Compared to static swapped queries, the queries in the QACA block are dynamic and exhibit better adaptability. Moreover, inspired by the powerful semantic understanding of language-vision model, we design a Text-Conditional Batch Normalization (TCBN) block which utilizes the text embeddings as a guidance to modulate image features. By leveraging high-level language semantics, the TCBN block flexibly balances spatial and spectral features, thereby effectively enhancing visual quality. Extensive experiments on both reduced-resolution and full-resolution datasets acquired by different satellites demonstrate the superiority of Text-QAViT over state-of-the-art methods.