CAFTrans: Content-Aware Fusion Transformer for ground-based remote sensing cloud detection

Recently, Transformer structures have been applied to ground-based remote sensing data to detect clouds, and the vanilla self-attention as the core component of Transformer utilize all tokens in the process of modeling long-range dependencies, resulting in involving less semantically correlated tokens. In this paper, we propose a novel Transformer network named Content-Aware Fusion Transformer (CAFTrans) for the ground-based remote sensing cloud detection task, which could effectively select the relevant tokens according to the content information of cloud sample. To this end, we propose the Content-Aware Selection Attention (CASA) in the Transformer encoder where we first construct the token-to-token similarity matrix with a learnable weight matrix and then dynamically select the tokens with high semantic relevance for each query according to the similarity matrix. Meanwhile, we introduce the Multi-Scale Fusion Mechanism (MSFM), which is built upon CASA and designed to capture long-range dependencies across multiple feature scales. To facilitate model training and evaluation, we present the Large-Scale Occluded Cloud Detection Dataset (LOCDD)—the first ground-based remote sensing dataset to consider three categories: clouds, sky, and occlusions. Comprehensive quantitative results and qualitative visualizations on the LOCDD dataset demonstrate the robust performance of the proposed CAFTrans model. The source code is freely available at: https://github.com/shuangliutjnu/CAFTrans . • We propose a new Transformer model for ground-based cloud detection. • A content-aware attention mechanism improves the token selection during learning. • A multi-scale fusion method captures scale variations in cloud samples. • We release the three-category large-scale dataset with clouds, sky, and occlusions.