A real-time semantic 3D vineyard mapping and fruit localization system for yield estimation in dynamic vineyards

• Proposed a real-time 3D mapping and grape localization system for dynamic vineyard. • A lightweight EFSegNet network enables high-precision grape bunch segmentation. • Proposed a three-stage filtering method for point cloud optimization • Implement DBSCAN clustering and PCA for fruit localization and yield estimation. In precision viticulture, autonomous systems are increasingly deployed for inspection and yield estimation. However, mapping 3D vineyard environments and monitoring fruit distribution presents challenges due to fruit occlusion, moving objects, and unstructured backgrounds. To address these issues, this study presents a semantic 3D mapping and fruit localization system designed for yield estimation in dynamic environments. The proposed framework integrates the ORB-SLAM3 backend with a lightweight semantic segmentation model, named Edge Fusion Segmentation Network (EFSegNet). Derived from the PIDNet baseline, EFSegNet introduces a Gradient Feature Fusion Block (GFFM) and a Deformable Spatial Attention (DSA) module. These components are designed to refine fruit boundaries and separate overlapping grape bunches, a requirement for individual counting. The generated semantic masks guide the removal of dynamic feature points to maintain map consistency, while enabling semantic mapping of fruit clusters. Additionally, a three-stage point cloud filtering strategy is applied to refine the global map, followed by DBSCAN clustering and PCA-based modeling to extract the 3D location and quantity of grape bunches. Validation experiments in a real-world dynamic vineyard indicate that the system operated at 23.1 FPS, with an average 3D localization error of 21.02 mm. The average counting error was 5.95%, demonstrating the system’s applicability for automated yield estimation.