Plant stem occlusion inpainting with Deep Reinforcement Learning

Growth monitoring of tomato plants in large greenhouse environments is critical for quality and efficient production. Stem diameter and elongation are key phenotypic traits for plant growth monitoring. Traditional methods, however, rely on manual operations, which are time-consuming and labor-intensive and do not apply to large-scale greenhouses. Currently, automated image-based methods exemplified by three-dimensional (3D) point cloud technology are among the preferred solutions. Nevertheless, the occlusion of plant structures during the information acquisition process is challenging for practical applications. To address this challenge, this study proposes a novel method for plant stem occlusion inpainting using Deep Reinforcement Learning (DRL). Unlike most existing 3D reconstruction approaches that require depth data from multiple viewpoints, our solution captures 3D point cloud data from a single direction. The DRL model is applied to inpaint the incomplete stem for accurate stem reconstruction and phenotypic measurements. Specifically, our approach consists of two parts, structural completion and stem diameter completion. First, we extract the point cloud of incomplete stems from the RGB-D camera data. Second, we obtain the spatial structure of the stems by inpainting the 3D stem centerline with the DRL model. Finally, we add shape features (stem diameters) by inpainting the two edge lines of the stem occlusion part with the DRL model. For stem inpainted 3D point cloud data, we conducted validation experiments by measuring several commonly used stem phenotypic traits in tomato plants, including stem diameter, stem length, and stem inclination. The experimental results show that the Mean Absolute Percentage Error (MAPE) of the occluded main stem diameter is 9.7%, stem length is 5.7%, and tilt angle is 1%. For the occluded branch stem, the MAPE of stem diameter is 23.1%, stem length is 7.9%, and tilt angle is 1.5%. The accuracy of these measurements for occluded stems is acceptable compared to that obtained from 3D point clouds of unoccluded stems. This highlights the significant potential of using DRL to effectively inpaint occluded 3D point cloud data of plants. • A tomato stem measurement method for single angle point cloud collection. • Deep Reinforcement Learning conducts point cloud inpainting for occluded stems. • The occlusion inpainting is in two steps, structure completion and diameter completion. • The proposed method is validated by phenotypic measurements such as stem diameter and length.