A novel opto-tactile sensing approach to enhance the handling of soft fruit

• Integrated DIGIT tactile sensors with a Robotiq 2F-85 and a UR10e robotic arm. • Developed a vision-based pipeline for delicate deformation analysis in soft objects. • Achieved accurate detection of subtle deformations in rubber balls and strawberries. • Showcased vision-based tactile sensing for precision farming and smart agriculture. In agricultural settings, handling of soft fruit is critical to ensuring quality and safety. This study introduces a novel opto-tactile sensing approach designed to enhance the handling and assessment of soft fruit, with a case example of strawberries. Our approach utilises a Robotiq 2F-85 gripper equipped with the DIGIT Vision-Based Tactile Sensor (VBTS) and attached to a Universal Robot UR10e. In contrast to force-based approaches, we introduce a novel purely image-based processing software pipeline for quantifying localised surface deformations in soft fruit. The system integrates fast and explainable image processing techniques applying image differencing, denoising, K-means clustering for unsupervised classification, morphological operations, and connected components analysis (CCA) to quantify surface deformations accurately. A calibration of the image processing pipeline using a rubber ball showed that the system effectively captured and analysed the rubber ball’s surface deformations, benefiting from its uniform elasticity and predictable response to compression. As a soft fruit case example, the image processing pipeline was subsequently applied to strawberries, blueberries, and raspberries, demonstrating that the calibration parameters derived from the rubber ball could effectively assess surface deformations in soft fruits. Despite the complex, nonlinear deformation characteristics inherent to strawberries, blueberries, and raspberries, the pipeline exhibited robust performance, capturing and quantifying subtle surface changes. These findings underscore the system’s capacity for precise deformation analysis in delicate materials, offering major potential for further refinement and adaptation. This novel approach of proposing and testing an image processing pipeline lays the groundwork for enhancing the handling and assessment of materials with intricate mechanical properties, paving the way for broader applications in sensitive agricultural and industrial settings.