Sorghum segmentation and leaf counting using in silico trained deep neural model

Abstract This paper introduces a novel deep neural model for segmenting and tracking the number of leaves in sorghum plants in phenotyping facilities. Our algorithm inputs a sequence of images of a sorghum plant and outputs the segmented images and the number of leaves. The key novelty of our approach is in training the deep neural model. Manual annotations are tedious, and we have developed a procedural three‐dimensional (3D) sorghum model that provides detailed geometry and texture to generate photorealistic 3D models. The overall shape of the sorghum leaf geometry is determined by its skeleton, and it is detailed by a procedural model that varies its curvature, width, length, and overall shape. The color is determined by using a Monte Carlo path tracer. We mimic the illumination of the phenotyping facility and use reflectance and transmittance on sorghum surfaces to determine the color of the leaves. The 3D procedural model allows us to generate photorealistic and segmented images that we use to train a deep neural model. Our segmentation provides a mean intersection over union score of 0.51, resulting in leaf counting accuracy within the 90% confidence interval for the human counts.