Organ3DNet: A deep network for segmenting organ semantics and instances from dense plant point clouds

Worldwide food shortage has put the plant phenotyping research to the spotlight because phenotyping enhances crop yield under limited land use by accelerating the cycle of modern breeding. The prerequisite of organ-level phenotyping is the accurate segmentation of organs from crop point clouds. Though mainstream deep networks have reported satisfactory results on certain species, they usually require sampling each input point cloud to a fixed number before network learning. Most existing networks also recommend each input to contain less than 10,000 points, which may smooth the structural details and lead to deterioration of segmentation of small organs. Moreover, these methods are still struggling to face challenges such as segmenting crops with a large number of organs and scalability to multiple species. In this paper, we propose Organ3DNet—a novel deep-learning-based architecture tailored for organ segmentation on high-precision plant 3D data. By integrating a Sparse 3D Convolutional Network Backbone (S3DCNB) as encoder and a new Transformer Decoder part containing a cascade of Query Refinement Modules (QRM) and Mask Modules (MM), Organ3DNet begins with query points obtained with 3D Edge-preserving Sampling (3DEPS) and gradually refines those queries into masks to effectively represent different organ instances. A high-precision dataset containing 889 samples from five species is also provided in this study. In experiment on this dataset, our Organ3DNet outcompeted four networks including ASIS, JSNet, PlantNet, and PSegNet. On the organ semantic segmentation task, our method surpasses the second best JSNet by 2.10 % on F1 and 3.63 % on IoU; while on the instance segmentation task, Organ3DNet surpasses the second best PSegNet by large margins of 16.46 % on mCov and 13.44 % on mWCov, respectively. Validation tests also show that Organ3DNet performs well on downstream tasks and real agricultural scenarios. The dataset and code associated with Organ3DNet are open to the readers. • An open crop point cloud dataset including 5 different species with organ semantics and instance annotations. • Our Organ3DNet can perform organ-level semantic and instance segmentation tasks directly on dense 3D crops. • Organ3DNet integrates advanced computational modules, and outcompetes several state-of-the-art segmentation networks. • The segmented organs by Organ3DNet can be directly used for precise calculation of organ-level phenotypes.