A multimodal deep learning for rice crop health monitoring using vegetation index feature selection

Crop health monitoring is essential for sustaining agricultural productivity and ensuring stable production, particularly in major crop-producing regions that face challenges due to climate variability, resource constraints, and spatial heterogeneity. Traditional methods struggle to effectively integrate spatial heterogeneity and environmental variability from multi-source remote sensing (RS) data. To address this problem, the proposed work explores the role of satellite imagery in monitoring rice crop health with the help of Vegetation Indices (VIs) under the impact of changing weather conditions. Multi-temporal Sentinel-1 and Sentinel-2 satellite data were acquired from a rice cultivation field in Gorakhpur, Uttar Pradesh, India, during three Kharif growing seasons (2023-2025). In this work, we propose a supervised MI-based feature selection framework employing Otsu’s adaptive thresholding for selection of VIs that strongly determine the health of rice crop. To ensure data integrity, cloud-contaminated observations were systematically excluded through a two-stage cleaning process. Further, we propose a multimodal deep learning (DL) framework that fuses a Swin Transformer (Swin-T) backbone for spatial feature extraction from vegetation index imagery with a Multi-Layer Perceptron (MLP) for structured numerical modeling of meteorological variables. The proposed framework achieved 94.05% accuracy on an independent held-out test set and a mean accuracy of \(92.12 \pm 5.28\) % under 10-fold cross-validation, consistently outperforming competing hybrid deep learning baselines. These results highlight that integrating transformer-based spatial representations with environmental information provides an efficient approach for rice crop health classification. The proposed framework can be extended to multi-region crop monitoring by incorporating additional vegetation index modalities and diverse climatic conditions.