SpecGateNet: Spectral-Guided Fusion Network for Cloud and Cloud Shadow Segmentation in Optical Remote Sensing Imagery

Accurate segmentation of clouds and cloud shadows in optical remote sensing imagery is an indispensable preprocessing step for downstream land-surface observation tasks. However, existing methods strike different trade-offs between local detail preservation and global context modeling, yet they still exhibit two major limitations: insufficient interaction during the dual-branch encoding stage, and decoder-side multi-scale fusion that applies uniform operations to all spatial locations, making it unable to distinguish boundary regions, such as thin cloud boundaries and cloud shadow contours, from homogeneous regions, such as thick cloud interiors. To address these issues, we propose SpecGateNet, a fusion network guided by spectral information from a frequency-domain perspective. The core idea is that the Fourier amplitude spectrum of deep features summarizes how much energy is carried by different spatial frequencies, where high-frequency components typically correspond to rapidly varying regions such as boundaries and textures, whereas low-frequency components are more closely associated with smooth regions. Building on this insight, we use an amplitude-derived, phase-free spectral energy cue constructed from the amplitude spectrum, and integrate it with a learnable gating generation network to produce spatially adaptive fusion modulation signals. SpecGateNet is built upon a CNN-Swin Transformer dual-branch encoder and consists of three key components: a Cross-Feature Fusion module (CFF) that enables stage-wise bidirectional interaction, a frequency-domain dynamic filtering bottleneck that enhances global context through input-adaptive frequency-domain filtering, and a Spectral-Guided Fusion decoder (SGF) that generates spatial gates from the amplitude spectrum to adaptively balance high-level semantics and low-level details. Together, the bottleneck filter and SGF constitute a spectral utilization framework that operates at both the encoder and decoder sides. Using only RGB inputs without relying on infrared auxiliary bands, SpecGateNet achieves the highest mIoU among the compared general-purpose baselines under a unified experimental protocol on three public datasets, namely CloudSEN-12, SPARCS-Val, and 38-Cloud, with mIoU scores of 77.80%, 76.75%, and 93.30%, respectively, outperforming the second-best method by 1.33, 5.26, and 1.44 percentage points. Ablation studies confirm that the spectral modules account for 70.1% of the total performance gain, with SGF contributing the largest single-step improvement among individual modules (+1.22%).