Walnut tree 3D posture determination and positioning for vibration harvesting using AI-binocular vision

As a globally important crop, walnut has multiple values in terms of food security, ecology and economy. Its efficient harvesting technology is crucial to increase yield and reduce labor costs, but the existing vibratory harvesting equipment relies too much on manual adjustment of operating parameters, resulting in low efficiency, poor adaptability, and difficulty in adapting to the characteristics of different tree trunks. This study innovatively proposes a three-dimensional trunk localization and attitude estimation method based on two-dimensional deep learning image processing and binocular vision to help upgrade the intelligent walnut vibration harvesting technology. In this study, a trunk bounding box-based SAM model is used to generate weakly supervised labels, train a lightweight YOLOv11 segmentation model, and extract the target trunk region from a 3D point cloud via a 2D mask, which is combined with skeleton analysis to realize trunk localization and pose estimation. The experimental results show that after the introduction of 5-meter depth threshold filtering, the 2D segmentation AP50 reaches 99.5% (the model is only 6MB), the average absolute error of 3D localization is <4cm, and the attitude estimation error is 1.2°, which verifies the high accuracy and embedded deployment feasibility. The weakly supervised strategy of integrating SAM and YOLOv11 effectively solves the problems of high annotation cost and model lightweighting, while the integration of binocular vision and 2D segmentation circumvents the limitations of traditional 2D attitude estimation without complex 3D point cloud processing. The method can support the intelligent clamping decision of walnut vibration harvesting equipment.