A Multimodal Deep Learning Framework for Plant Disease Classification Using Pseudo-Temporal Leaf Image Sequences and Climatic Factors

Plant diseases are a major threat to global agricultural productivity and food security, demanding accurate and timely prediction for effective crop management. Recent advances in deep learning have enabled automated disease detection using plant leaf images and environmental data. However, most existing approaches are limited to analysis of static images and therefore provide limited scope for evaluating sequential visual representation learning associated with disease symptoms. To address these limitations, this study proposes a hybrid multimodal deep learning framework integrating Convolutional Neural Networks (CNN) and Video Swin Transformers for enhanced plant disease classification. The CNN module extracts discriminative spatial features from individual leaf images, while the Video Swin Transformer captures inter-frame dependencies across sequential images, enabling sequential visual representation learning. Additionally, key climatic factors, including temperature, humidity, rainfall, and soil moisture, are incorporated through a dedicated feature extraction branch to account for environmental impacts on disease development. These learned visual and climate embeddings are fused via a lightweight method of feature-level fusion that uses concatenation of the two embeddings followed by multilayer perceptron classification. The proposed framework is capable of performing fine-grained classification on 14 different classes of plant diseases and healthy plants, thus accurately detecting crop-specific diseases in various environmental conditions. In order to make the model more transparent, Explainable Artificial Intelligence (XAI) methods are used to explain which visual and environmental factors are responsible for model decisions. Besides, there is an agronomic rule-based interpretation module that helps to get explanatory insights into environmental features linked to disease prediction. Experimental results demonstrate superior performance, achieving 98.91% accuracy, 0.9905 precision, 0.9855 recall, and 0.9879 F1-score, validating the effectiveness of the proposed framework. Though this framework is based on pseudo-temporal images because real longitudinal disease data is limited, results show great potential for using sequential visual features and environment together for robust classification of plant diseases.