MKG-CottonCapT6: A Multimodal Knowledge Graph-Enhanced Image Captioning Framework for Expert-Level Cotton Disease and Pest Diagnosis

As one of the world’s leading cotton-producing countries, China frequently experiences severe yield reductions due to crop diseases and pest infestations, with losses often exceeding 20%. Although computer vision models can identify diseased plants, they currently fail to connect visual symptoms to the diagnostic reasoning process used by agronomists. This leads to text descriptions that ignore the biological causes of the damage. To fix this, we built Multimodal Knowledge Graph-Enhanced Cross Vision Transformer-18-Dagger-408 and Text-to-Text Transfer Transformer for Cotton Disease and Pest Image Captioning (MKG-CottonCapT6), a model that uses a local knowledge database to generate professional diagnostic reports from field images. The technical core consists of a Multimodal Knowledge Graph (MMKG) containing 14 types of entities (such as Pathogens and Control Agents) and 12 types of relations. We use a Cross Vision-Transformer-18-Dagger-408 (CrossViT) encoder to capture both the overall leaf shape and microscopic details of pests. Through a Visual Entity Grounding (VEG) module, the model maps visual features directly to specific triplets in the graph. These triplets are then turned into text sequences and fused with image data in a Text-to-Text-Transfer-Transformer (T5) decoder. To train the model, we collected a dataset of cotton images paired with expert descriptions of lesions, colors, and affected plant parts. Tests show that MKG-CottonCapT6 performs better than standard models, reaching an Information-based Metric for Image Captioning (InfoMetIC) score of 72.6%. Results prove that by using a specific alignment loss (Lalign), the model generates reports that correctly name the disease stage and recommend specific chemicals, such as Carbendazim or Triadimefon. This framework provides a practical tool for farmers to record and treat cotton diseases with high precision.