Dual-Detector Vision and Depth-Aware Back-Projection for Accurate Apple Detection and 3D Localisation for Robotic Harvesting

Accurate apple detection and precise three-dimensional (3D) localisation are essential for autonomous robotic harvesting in orchard environments, where occlusion, illumination variation, depth noise, and the similar colour appearance of fruits and surrounding leaves present significant challenges. This paper proposes a dual-detector vision framework combined with depth-aware back-projection to achieve robust apple detection and metric 3D localisation in real time. The method integrates the complementary strengths of YOLOv8 and Mask R-CNN through confidence-weighted fusion of bounding boxes and pixel-wise union of segmentation masks, producing stabilised two-dimensional (2D) apple representations under visually ambiguous conditions. The fusion results are converted into dense 3D representations through depth-guided projection within the camera coordinate system representing the visible fruit surface. A depth-consistency weighting strategy assigns higher influence to depth-reliable pixels during centroid computation, thereby suppressing noisy or occluded depth measurements and improving the stability of 3D fruit centre estimation, while local intensity normalisation standardises neighbourhood-level pixel intensities to reduce the impact of shadows, highlights, and uneven lighting, enabling more consistent segmentation and detection across varying illumination conditions. Experimental results demonstrate an accuracy of 98.9%, an mAP of 94.2%, an F1-score of 93.3%, and a recall of 92.8%, while achieving real-time performance at 86.42 FPS, confirming the suitability of the proposed method for robotic harvesting in challenging orchard environments.