Detect, segment, cluster: Apple localization for robotic harvesting in complex orchards

Efficient robotic apple harvesting hinges on robust 3D vision systems capable of accurately identifying and localizing fruits in complex orchard environments. While localization algorithms leveraging RGBD sensors and deep learning methods exist–through extrapolating 2D detections into 3D space, they often suffer from occlusions and noisy depth data, resulting in limited real-world accuracy. To overcome these limitations, we propose a unified 3D localization framework that seamlessly integrates vision-language model (VLM)-based apple detection, instance segmentation, and unsupervised point-cloud clustering. Our method first detects and segments apples in 2D images using a foundation VLM (e.g., Grounding-DINO) and a high resolution instance segmentation model. The identified apple pixels are then projected into 3D space, where we apply Density-Based Spatial Clustering of Applications with Noise (DBSCAN) to refine localization by filtering out noise and unreliable depth measurements. We benchmark our approach against traditional 2D-to-3D extrapolation and 3D localization techniques, demonstrating its superior robustness and precision under challenging conditions. Evaluated on a dataset of RGBD images containing 219 apples across indoor and outdoor environments, including both real and synthetic apples, our method achieves a mean absolute localization error of 7.7 mm—a 74.3% improvement over the best-performing baseline methods. Our approach has been deployed on an in-field robotic harvesting system, demonstrating real-time performance and strong robustness to occlusions and variable lighting—highlighting its practical value for autonomous fruit picking in unstructured orchard environments. • Robust 3D apple localization pipeline improves localization accuracy of occluded fruits. • Density-based clustering enables real-time noise filtering for robotic har- vesting. • Vision-language detection, segmentation, and clustering deployed on a field robot.