Soybean Lodging Classification and Yield Prediction Using Multimodal UAV Data Fusion and Deep Learning

UAV remote sensing is widely used in the agricultural sector due to its non-destructive, rapid, and cost-effective advantages. This study utilized two years of field data with multisource fused imagery of soybeans to evaluate lodging conditions and investigate the impact of lodging grade information on yield prediction. Unlike traditional approaches that build empirical lodging models using band reflectance, vegetation indices, and texture features, this research introduces a transfer learning framework. This framework employs a ResNet18 encoder to directly extract features from raw images, bypassing the complexity of manual feature extraction processes. To address the imbalance in the lodging dataset, the Synthetic Minority Over-sampling Technique (SMOTE) strategy was employed in the feature space to balance the training set. The findings reveal that deep learning effectively extracts meaningful features from UAV imagery, outperforming traditional methods in lodging grade classification across all growth stages. On the 65 days after emergence (DAE), lodging grade classification using ResNet18 features achieved the highest accuracy (Accuracy = 0.76, recall = 0.76, F1 score = 0.73), significantly exceeding the performance of traditional methods. However, classification accuracy was relatively low in plots with higher lodging grades (lodging grades = 3, 5, 7), with an accuracy of 0.42 and an F1 score of 0.56. After applying the SMOTE module to balance the samples, the classification accuracy in plots with higher lodging grades improved to 0.65, marking an increase of 54.76%. To improve accuracy in yield prediction, this study integrates lodging information with other features, such as canopy spectral reflectance, vegetation indices, and texture features, using two multimodal data fusion strategies: input-level fusion (ResNet-EF) and intermediate-level fusion (ResNet-MF). The findings reveal that the intermediate-level fusion strategy consistently outperforms input-level fusion in yield prediction accuracy across all growth stages. Specifically, the intermediate-level fusion model incorporating measured lodging grade information achieved the highest prediction accuracy on the 85 DAE (R2 = 0.65, RMSE = 529.56 kg/ha). Furthermore, when predicted lodging information was used, the model’s performance remained comparable to that of the measured lodging grades, underscoring the critical role of lodging factors in enhancing yield estimation accuracy.