3D plant segmentation: Comparing a 2D-to-3D segmentation method with state-of-the-art 3D segmentation algorithms

Plant measurements are crucial to determine which plants grow optimal under certain conditions. These measurements can be done by hand, or automated using cameras, also known as image-based plant phenotyping. These images can be used to create point clouds to measure plant traits in 3D. To extract plant traits, accurate segmentation is crucial. Most point cloud segmentation methods rely on 3D segmentation algorithms. These algorithms are not as advanced and developed as 2D algorithms. In addition, 2D neural networks are pre-trained on large diverse datasets. In our work, it was therefore hypothesised that segmentation of point clouds using projection-based methods can obtain a higher accuracy than voxel or point-based algorithms. To test this hypothesis, a 2D-to-3D reprojection method was developed and compared with three state-of-the-art 3D segmentation algorithms; Swin3D-s, Point Transformer v3 and MinkUNet34C. The 2D-to-3D method segmented images using Mask2Former, reprojected the predictions to the point cloud, and used a majority vote algorithm to merge multiple predictions. All algorithms were trained and tested to segment 3D point clouds into leaves, main stem, side stem, and pole. There was no significant difference between the 2D-to-3D, Swin3D-s and Point Transformer v3 algorithm, indicating that state-of-the-art voxel or point-based methods perform similar than our projection-based method. However, the 2D-to-3D method had a higher performance by including virtual cameras and it had a higher training efficiency. With only five annotated plants, a similar performance was obtained than training Swin3D-s on 25 plants indicating the added value of the developed pipeline.