Visual Question Answering of Plant Disease with BLIP and FLAN T5 Enhancement

Early and accurate diagnosis of plant diseases is crucial for ensuring crop health and optimizing agricultural productivity. Traditional approaches often require expert intervention, which may not always be accessible to farmers, while existing automated systems typically provide only basic classification without interactive diagnostic capabilities or contextual explanations. To address this gap, this study proposes an AI-driven Plant Disease Visual Question Answering (VQA) system that enables users to interactively obtain diagnostic insights from plant images. The methodology utilizes Bootstrapped Language Image Pretraining (BLIP) as the foundational VQA model, which is trained on structured, domain-specific multi-question annotations. To improve the interpretability of the model and the fluency of its responses, this study incorporates Hugging Face’s Fine-tuned LAnguage Net (FLAN) T5 (Text-To-Text Transfer Transformer) for answer enhancement. Furthermore, it establishes a tailored knowledge base for each category of plant disease, systematically assembled to boost the contextual precision and depth of the responses—an innovative aspect of this study. Fine-tuning was performed using the PlantDoc dataset, which includes 13 different disease categories and 17 plant species. Evaluations based on BLEU scores resulted in a moderate score of 0.2671, while qualitative assessments reveal a high level of domain-specific accuracy, underscoring the model’s resilience despite computational limitations. This study’s findings underscore the potential of multimodal VQA in the realm of plant disease diagnosis, setting the stage for scalable and intelligent decision-support systems in the agricultural sector.