SkyMamba: Integrating Transformer and State-Space Model for UAV Remote Sensing RGB-D Images Semantic Segmentation

Uncrewed aerial vehicles (UAVs) aerial remote sensing (RS) images offer higher spatial resolution compared with satellite RS images, providing more detailed scene information. The flexibility of UAV-mounted sensors allows for the simultaneous acquisition of multimodal data, including RGB images and depth maps. Depth information, serving as a supplementary cue, has been extensively utilized in semantic segmentation research. Nevertheless, the current investigations on red, green, blue-depth (RGB-D) image semantic segmentation predominantly concentrate on indoor or road scenarios. In UAV RS scenes, there exist numerous objects featuring similar visual texture manifestations. The introduction of depth geometric information can efficaciously enhance the semantic segmentation accuracy of complex ground objects. Concurrently, high-resolution RS images also present challenges to the modeling of global information in large-scale space. To address these challenges, we propose the first framework for semantic segmentation of RGB-D images in UAV RS, namely, SkyMamba. This framework effectively integrates visual texture information and geometric depth information. Simultaneously, it leverages the long-sequence modeling capability of the state-space model to efficiently capture global context information in the RS space, achieving efficient semantic segmentation of RS scenes. Furthermore, we introduce a novel benchmark dataset for semantic segmentation of UAV RS RGB-D images, named comprehensive urban-scale depth, which is divided into two versions based on coarse-grained and fine-grained semantic annotation strategies to provide diverse challenges. Experimental results demonstrate that the proposed method exhibits state-of-the-art performance on both versions of the dataset, particularly in handling imbalanced scene semantic class distributions. The code for this article is available at the following URL: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/CapyLin-G/CUS-Depth</uri>