Utilizing Cross Layer Attentions for Semantic Segmentation of Small Objects

Segmenting small objects poses significant challenges in semantic segmentation tasks. This is because the downsampling process employed by many current deep learning networks tends to suppress the features of small objects, making them difficult to be retained in the encoded latent space. To mitigate this issue, we propose the Cross Layer Attention (CLA) encoder/decoder block, which aims to correlate informative features between deep feature layers. CLA calculates the correlation across different feature layers in the encoder and then subsequently integrating this information is integrated into the Feed-Forward Network (FFN) decoder, thus forming a dual decoding process. The designed CLA is able to preserve the features of objects at different positions, so that it can be used to enhance the feature of a small object during the upsampling process. The proposed CLA is experimented with the ADE20K public dataset using the Segformer <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">–</inf> B0 and B1 as the backbone network, and the results are imrpoved by 1.37% and 2.38% in mean Intersection over Union (mIoU) compared to the original SegFormer <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">–</inf> B1 and B0 model respectively. Furthermore, for objects which often appear smaller than 32×32 pixels in the validation set, the mIoU is also improved by 3.31% and 7.02%. The experiments demonstrates that the proposed CLA enhances the overall semantic segmentation results, particularly for small objects.