A Framework for Agricultural Intelligent Analysis Based on a Visual Language Large Model

Smart agriculture has become an inevitable trend in the development of modern agriculture, especially promoted by the continuous progress of large language models like chat generative pre-trained transformer (ChatGPT) and general language model (ChatGLM). Although these large models perform well in general knowledge learning, they still have certain limitations and errors when facing agricultural professional knowledge about crop disease identification, growth stage judgment, and so on. Agricultural data involves images and texts and other modalities, which play an important role in agricultural production and management. In order to better learn the characteristics of different modal data in agriculture, realize cross-modal data fusion, and thus understand complex application scenarios, we propose a framework AgriVLM that uses a large amount of agricultural data to fine-tune the visual language model to analyze agricultural data. It can fuse multimodal data and provide more comprehensive agricultural decision support. Specifically, it utilizes Q-former as a bridge between an image encoder and a language model to achieve a cross-modal fusion of agricultural images and text data. Then, we apply a Low-Rank adaptive to fine-tune the language model to achieve an alignment between agricultural image features and a pre-trained language model. The experimental results prove that AgriVLM demonstrates great performance in crop disease recognition and growth stage recognition, with recognition accuracy exceeding 90%, demonstrating its capability to analyze different modalities of agricultural data.