Finger vision enabled real-time defect detection in robotic harvesting

• Proposed an eye-in-finger configuration for fruit inspection, first in the field. • First pre-harvest fruit inspection method combined with robotic harvesting. • Achieves 98 % precision of apple defect detection. • Achieves real-time defect detection using under 20% of Jetson Orin’s capacity. This study introduces a pioneering solution for pre-harvest fruit defect detection in the agricultural industry. Manual inspection by human pickers is inconsistent and labour-intensive, while traditional automated methods require harvested fruits to be placed in controlled environments, leading to wasted time and resources on picking and transporting defective fruits. This research presents a novel, finger vision-enabled, real-time defect detection method for robotic harvesting. Based on a comprehensive analysis of camera configurations for fruit inspection, an eye-in-finger configuration is proposed for the first time in the field, a versatile finger is created that transforms a robotic gripper from merely a grasping tool into a powerful inspection device, potentially achieving 81 % and 89 % higher fruit surface coverage compared to the eye-on-base and eye-in-hand configurations, respectively. A prototype with four low-cost, off-the-shelf cameras embedded in four fingers was built with a gripper to perform rotational inspection around the target fruit. Four YOLOv8 model variants were leveraged and trained to identify defect features from the images collected by the eye-in-finger cameras, demonstrating a precision of up to 98 %. A real-time detection algorithm incorporating YOLOv8-p2 was developed and successfully validated in the field with a latency of only 39.2 ms and a model size of 21.4 MB, while utilising less than 20 % of the central computer’s capacity. The successful field demonstration of the proposed finger vision-based defect detection method indicates a significant potential of automating pre-harvest inspection, reducing labour costs, and mitigating the time and resources spent on post-harvest sorting.