Assessing robustness and transferability of image‐based semantic segmentation models for major field crops

Abstract Plant breeding is essential for crop improvement, yet progress is often hindered by slow, laborious, and subjective field phenotyping methods. High‐throughput phenotyping (HTP), particularly image‐based methodologies powered by machine learning, offers a pathway to overcome these limitations. However, achieving robustness and generalization when analyzing diverse genotypes within a crop and across reproductive stages remains challenging and can affect model performance and the accurate extraction of phenotypic features. This study evaluated the performance of semantic segmentation models across a diverse panel of genotypes and distinct crop reproductive stages, using wheat ( Triticum aestivum L.), sorghum ( Sorghum bicolor L.), and corn ( Zea mays L.) as case studies. The primary objectives were to analyze (i) the overall prediction performance on the aggregated dataset for each crop, (ii) the stratified performance by genotype and collection date, and (iii) the temporal and genotypic transferability across growth stages and unseen genotypes. Four distinct smartphone cameras were used to collect images of the reproductive structure across crop growth stages (different collection dates) from 160 corn, 80 sorghum, and 40 wheat genotypes. The total number of images per crop was 2000 for wheat, 4000 for sorghum, and 3840 for corn. Five semantic segmentation models were tested in this study—DeepLabv3+, MaskFormer, SegFormer, SegNet, and U‐Net—using the images and respective binary masks for training and testing. The SegFormer model achieved the highest intersection over union (IoU) values for corn (0.90) and sorghum (0.92), while the U‐Net model performed best for wheat (0.89). A minor performance decline, with IoU differences up to 0.1, was observed when testing the same model across different genotypes. However, the temporal transferability drops up to 0.5 IoU when training and inferring on different crop growth stages. The main reason for those changes may lie in the natural color and organ architecture temporal changes between the trained and tested datasets when transferring the models across growth stages. These results highlight the urgent need to prioritize robustness and transferability when developing reliable in‐field HTP methodologies.