Maize phenological stage recognition via coordinated UAV and UGV multi-view sensing and deep learning

Crop phenological stages, marked by key events such as germination, leaf emergence, flowering, and senescence, are critical indicators of crop development. Accurate, dynamic monitoring of these stages is essential for crop breeding management. This study introduces a novel multi-view sensing strategy based on coordinated unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs), designed to capture diverse canopy perspectives for phenological stage recognition in maize. Our approach integrates multiple data streams from top-down and internal-horizontal views, acquired via UAV and UGV platforms, and consists of three main components: (i) Acquisition of maize canopy height data, top-of-canopy (TOC) digital images, canopy multispectral images, and inside-of-canopy (IOC) digital images using a UAV- and UGV-based multi-view system; (ii) Development of a multi-modal deep learning framework, MSRNet (maize-phenological stages recognition network), which fuses physiological features from the UAV and UGV sensor modalities, including canopy height, vegetation indices, TOC maize leaf images, and IOC maize cob images; (iii) Comparative evaluation of MSRNet against conventional machine learning and deep learning models. Across 12 phenological stages (V2–R6), MSRNet achieved 84.5 % overall accuracy, outperforming conventional machine learning and single-modality deep learning benchmarks by 3.8–13.6 %. Grad-CAM visualizations confirmed dynamic, stage-specific attention, with the network automatically shifting focus from TOC leaves during vegetative growth to IOC reproductive organs during grain filling. This integrated UAV and UGV strategy, coupled with the dynamic feature selection capability of MSRNet, provides a comprehensive, interpretable workflow for high-throughput maize phenotyping and precision breeding. • Multi-view UAV and UGV imagery used for maize phenological stage recognition. • MSRNet fuses features from CH, VIs, TOC leaf images, and IOC maize cob images. • MSRNet shifts focus from leaf to cob images in reproductive stages. • Grad-CAM reveals leaf/cob attention aligns with phenology.