Convolution and Transformer based hybrid neural network for Road Extraction in Remote Sensing Images

Road extraction in remote sensing images has a wide range of applications and is influential in the fields of automatic driving, city planning, navigation, etc. However, there are many shadows, buildings and trees on remote sensing images, which adds a lot of difficulties to extracting roads. In this paper, a hybrid neural network (HCTRNet) based on Convolution and Transformer is proposed for road extraction in remote sensing images. The HCTRNet backbone network has a pair of symmetric encoders and decoders, in which the main feature extraction module is HCTRNet Block. A new Hybrid Token Mixer (HT-Mixer) module is used in HCTRNet Block. HT-Mixer can dynamically utilise the global and local information of the image to incorporate the potential of large receptive fields and strong inductive bias without affecting the input dependency, which can largely improve the model's representation learning capability. After the decoder, this paper designs an Adaptive Feature Fusion Head module (AFFH) to integrate feature information from different scales in the feature layers in the decoding layer to enhance the robustness of the network for better road extraction. Experiments show that HCTRNet performs much better than current state-of-the-art algorithms on the publicly available Istanbul City Roads dataset, improving 10.04% in terms of IoU. The source code is publicly available at https://github.com/whq-27/HCTRNet.