A scalable and efficient UAV-based pipeline and deep learning framework for phenotyping sorghum panicle morphology from point clouds

Sorghum canopy architecture in field trials is determined by various phenotypic traits, such plant and panicle count, leaf density and angle and panicle morphology, and canopy height. These traits together affect light capture and biomass production as well as conversion of photosynthates to grain yield. Panicle morphology exhibits considerable variation as influenced by genetics, environmental conditions and management practices. This study presents a framework for the 3D reconstruction of sorghum canopies and phenotyping panicle morphology. First, we developed a scalable, low-altitude Unmanned Aerial Vehicle (UAV)-based protocol that leverages videos for efficient data acquisition, combined with Neural Radiance Fields (NeRF)s to generate high-quality 3D point cloud reconstructions of sorghum canopies. Next, a 3D model was built to simulate 3D sorghum canopies to create annotated datasets for training deep learning-based semantic segmentation and panicle detection algorithms. Finally, we propose SegVoteNet, a novel multi-task deep learning model that integrates VoteNet and PointNet++ within a shared backbone architecture. Designed for semantic segmentation and 3D detection on pure point cloud data, SegVoteNet incorporates a voting and sampling module that leverages segmentation results to optimize object proposal generation. SegVoteNet is robust, achieving 0.986 Mean Average Precision (mAP) @ 0.5 Intersection Over Union (IOU) on synthetic datasets, and 0.850 mAP @ 0.5 IOU on real point cloud datasets for sorghum panicle detection, without fine-tuning. This set of pipelines provides a robust scalable method for phenotyping sorghum panicles in field trials in breeding and commercial applications. Further work is developing a capability to estimate grain number per panicle, which would provide breeders with additional phenotypes to select.