D-SSSS: an Information Perception Algorithm for Apple Fruit Trees via Semi-supervised Semantic Segmentation and Depth Information Blending

In the natural environment, the distribution of apples and branches in apple fruit trees is complex, and manual annotation is extremely difficult, resulting in fewer semantic segmentation studies for apple fruit trees with less promotion and application. To address this issue, we have proposed the semantic segmentation model D-SSSS (Depth information fused in Semi-Supervised Semantic Segmentation) for apple fruit trees based on semi-supervised learning. In the encoder, we add the cross-modal feature rectification module and the feature fusion module to fully utilize the spatial information in the depth image and enhance the generalization ability of the model. At the same time, in order to further improve the prediction ability of the model, in our study, the feature fusion module is used in the decoder to fuse the multi-scale features of the encoder to achieve efficient utilization of features. The semantic memory module in the first step of the decoder adopts the cross-attention module, whose role is to comprehensively utilize the unlabeled feature information to generate the semantic segmentation result of the labeled image during the semi-supervised learning training process. Experimental results show that the D-SSSS model can comprehensively utilize the spatial features in the depth image to increase the mIoU by 2.57%, and can better predict the branch category. Compared with other semi-supervised semantic segmentation models, the model proposed in this study can achieve better semantic segmentation of fruit trees and show stronger generalization ability.