Multi-Modal Working Environment Perception for Autonomous Excavation of Mining Electric Shovel Based on Parallel Neural Network

Environment perception is a crucial technology of the super large type of intelligent electric shovel in open-pit mining for realizing automatic excavation. However, poor field view, variable lighting, and high-density dust at construction sites lead great technological challenges beyond the current human manipulation capability. The article proposes a multimodal semantic segmentation method based on LiDAR-camera fusion to classify each element of the construction site and accurately segment the ore piles. Specifically, the correspondence between point cloud points and pixels through perspective projection is established, and a parallel feature fusion model (PFFM) and a minimum cost function for feature extraction and optimization is built. Subsequently, radial basis function interpolation is used to reconstruct the surfaces of the segmented ore piles. Moreover, the excavation trajectory of electric shovel is optimized based on the reconstructed surface. The average segmentation accuracy and the per-frame segmentation time of PFFM reach 95.72% and 0.193s, respectively. Simulations and experiments show that trajectory planning based on the proposed method can significantly improve the operation performance.