Multimodal learning on RGB-D image for precise litchi phenotyping and weight estimation

Accurate measurement of key phenotypic traits, including the horizontal and vertical diameters, the weights of both fruit and pit, is essential for the selection of elite litchi cultivars and the advancement of breeding research. Manual measurement, however, is laborious, inefficient, and subjective, highlighting the urgent need for automated and precise phenotyping tools. Unlike apples, mangoes, and grapes, litchi combines a spiny, highly variable pericarp (heterogeneous areoles/tubercles across cultivars) with diverse seed morphology (including irregular, wrinkled aborted seeds), thereby increasing the difficulty of semantic segmentation and biasing diameters and weight estimation. This study presents LitchiPhenoNet, a multimodal learning framework for litchi phenotypic analysis that employs a dual-branch architecture integrating RGB (color/texture) and depth (spatial/structural) information. Experiments were conducted on an RGB-D dataset comprising 1,198 image pairs (1280×720) across 10 cultivars, using a stratified train/test split of 958/240 pairs by cultivar. To address inherent semantic and scale inconsistencies between modalities, the framework incorporates the RD-Fusion module for precise cross-modal feature extraction, improving robustness under complex and variable pericarp surfaces. Comparative experiments show that LitchiPhenoNet consistently outperforms leading YOLO-based models, achieving millimeter-level diameter estimation with coefficients of determination approaching 0.98 and mean errors within 2 mm. For weight estimation, gram-level precision is attained across whole fruit, pit, and pulp, with coefficients of determination up to 0.98 and mean errors comparable to repeated manual measurements. By handling fine-scale surface relief and cross-cultivar variability, the framework is readily extensible to other textured fruits and scalable for high-throughput phenotyping in breeding programs. Collectively, these results demonstrate that LitchiPhenoNet provides an efficient, reliable, and accurate solution for quantifying litchi phenotypic traits, substantially advancing the objectivity and efficiency of phenotypic analysis and breeding selection.