Integrating Visual and Environmental Modalities for Rice Leaf Diseases Classification and its Severity Prediction with Dual-Output CNN

Various living and non-living factors can significantly decrease the rice yield. Existing approaches to deep learning often fail to consider the influence of environmental factors on disease and severity classification. A multimodal deep learning model for predicting disease (10 classes) and severity (3 levels) of rice leaves continuously through a combination of visual disease features and environmental factors. The framework uses a dual-branch architecture: (i) a dual Convolution Neural Network (CNN)-based visual encoder to extract the features of the lesions, and (ii) a Multilayer Perceptron (MLP)-based environmental encoder to model the environmental conditions in the field. Fused images using cross-attention module, followed by dual head outputs for disease and severity prediction. Gradient-weighted Class Activation Mapping (Grad-CAM) visualization is used for the interpretation of lesion-focused decisions. The model is trained using multi-task loss and tested using five-fold cross-validation. This model obtained 97.9% accuracy, 97.6% macro-F1, and Cohen’s κ = 0.974 for disease classification, outperforming state-of-the-art baselines including ResNet50 (93.5%), EfficientNet-B4 (95.5%), and Swin Transformer (96.2%). For severity prediction, the results were 95.3% accuracy and a macro-F1 of 95.2% for the model. The addition of environmental features increased the accuracy by + 4.5% compared to the visual-only model. Grad-CAM confirmed that predictions were based on lesion regions associated with leaf disease. In future research, transformer-based multimodal fusion will be studied, real-time IoT-based environmental sensing will be added, and validation products for multi-location and multi-crop datasets will be studied.