A multisource image fusion framework for visual perception in orchards using a graph convolutional neural network and attention mechanisms applied to target detection

The accurate localization of fruit targets by visual perception systems is the key to achieving efficient automated picking in complex scenes in orchards. The fusion of multisource images originating from visual perception systems integrates complementary information from these modalities to produce fused images with both rich texture details and prominent targets, improving the image quality of visual perception systems and effectively enhancing the precise localization ability of target fruits. Existing fusion methods, such as convolutional neural networks and transformers, can only operate on regular Euclidean data and cannot handle graph-structured data in non-Euclidean space. Owing to the prevalent node dependency relationships in graphs, modelling the relationships between pixels in graphs helps to more accurately understand the structure and features in the image. Therefore, we present a multisource image fusion framework using a graph convolutional neural network and attention mechanisms and apply it to apple target detection. First, an encoder with a feature extraction module—a graph branch with a graph convolutional neural network and a graph attention mechanism jointly implemented with a channel‒spatial attention branch—and a multiscale processing module for multiscale feature extraction is constructed. Afterwards, a fusion strategy incorporating spatial attention is employed to enhance the model’s representation power of important image features. Finally, the fused images are generated using the decoder based on the nest connections. The results of the experiments indicate that the proposed approach can better retain the texture information of the fruit target region and increase the prominence of the target region. Furthermore, applying our fusion approach to apple target detection effectively improves the detection accuracy. Specifically, the mean average precision (mAP50) of the detection model increases from 65.4% of the visible image to 80.1% of the fused image generated by our approach. The source code is publicly available at: https://github.com/Zhaichenxi/GCN_AM.