Wheat Disease Monitoring Based on Multimodal Large Models

Wheat disease detection can reduce grain losses. Currently, wheat disease detection is conducted under specific names and fails to form a closed-loop operation of “Perception-Recognition-Forecast-Control”. To address this issue, we have applied a wheat disease monitoring method based on multi-modal large model. Firstly, the image is processed and divided into a certain number of patches which are input into the visual encoder. Simultaneously, the Knowledge text sequence of wheat disease is input into the text encoder. Then, the Q- Former method is used to fuse the visual features of the disease with the knowledge text features. Finally, knowledge question-answering for wheat disease images was realized. Experiments on our wheat disesase dataset results demonstrate that our method achieves the performance of 89.7<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">%</sup>, 83.2<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">%</sup>average recall, 34.6%,36.8% average accuracy and 49.8%, 49.9% average F1 scores. Our method could apply on the practical Integrated Disease Management (IDM).