3D pose estimation of strawberry using 2D keypoint detection and regression

Strawberry harvesting is a repetitive and labor-intensive task, with yield quality heavily dependent on the operator’s skill. To address labor shortages, improve productivity, and ensure consistent harvest quality, the development of harvesting robots is essential. In particular, implementing an autonomous harvesting system requires advanced perception technology capable of accurately identifying the position and condition of target fruit. However, simple object detection alone cannot provide sufficient kinematic information for automation, making keypoint detection of strawberries necessary. This study proposes a 3D pose estimation pipeline that combines a deep learning based keypoint detection model with a multi-layer perceptron (MLP) regression model to provide kinematic information applicable to strawberry harvesting robots. The centers of the fruit, calyx, and pedicel were defined as keypoints, with their 2D positions detected and corresponding depth values predicted using MLP regression. Performance evaluation included percentage of correct keypoints (PCK) for 2D accuracy, and mean per joint position error (MPJPE) and PCK for 3D depth estimation. Experimental results showed average 2D errors of mean PCK@5 pixels of 77.17%. for depth estimation, the average MPJPE was 16.63 mm and PCK@50 mm reached 91.3%. The proposed pipeline demonstrated stable detection of keypoint locations while preserving the overall 3D structure, indicating its potential contribution to the development of perception technologies for autonomous strawberry harvesting.