Semantic segmentation for virtual-real fusion data processing in nonferrous metal process industry

By combining high-precision virtual models and actual data, virtual-real fusion technology can significantly improve the production efficiency, quality control and safety of non-ferrous metal process industry, and optimize the production process. In the data processing of virtual-real fusion, environmental understanding is the basis for the correct interaction between virtual objects and the real world in the virtual-real fusion. Virtual-real convergence, especially in Augmented Reality (AR), often requires advanced image understanding techniques to improve user experience and provide more interactivity. Semantic segmentation plays a key role in this process, so this paper proposes a semantic segmentation method that uses RGB and point cloud images as well as deep image fusion to solve the heterogeneous problem of multi-source data in virtual-real fusion data processing, and improve the ability of machine vision system to understand the real world environment in virtual-real fusion. Firstly, Agent Attention is introduced for the decoder stage processing features, which combines the advantages of traditional attention and linear attention. Secondly, a feature fusion method is introduced for adaptive fusion of features between various stages. Experiments were carried out on the NYU Depth Dataset V2 and KITTI-360 data sets, and the segmentation results of the proposed method were 53.7% and 56.7% respectively on mIoU. In addition, the method is applied to the field of ore segmentation, and the mIoU reaches 81.7%, which also achieves good segmentation effect.