Vision Meets Language: A RAG-Augmented YOLOv8 Framework for Coffee Disease Diagnosis and Farmer Assistance

Abstract Timely and accurate identification of crop diseases is a critical challenge in precision agriculture, as delayed or incorrect diagnosis often leads to excessive pesticide use, yield loss, and environmental degradation. Existing vision-based disease detection systems primarily focus on classification accuracy but lack interpretability and decision support, while large language models (LLMs) suffer from hallucinations when deployed without domain grounding. To address these limitations, this paper proposes a knowledge-grounded vision–language framework that connects YOLOv8-based disease detection with a Retrieval-Augmented Generation (RAG) pipeline and a Large Language Model (LLM). The framework transforms structured visual detection outputs into retrieval queries that guide knowledge-grounded reasoning, enabling fact-consistent explanations and disease-specific remediation recommendations while mitigating hallucinations. Experiments conducted on the BRACOL coffee leaf dataset demonstrate that the proposed system achieves strong detection performance using YOLOv8n, attaining an overall [email protected] of 0.792 and [email protected]:0.95 of 0.523 on the original annotations. Compared to refined annotations, the original dataset yields consistent performance gains across all disease classes, highlighting the sensitivity of detection models to annotation strictness. Furthermore, a Non-Maximum Suppression (NMS) based ensemble of multiple YOLO variants improves robustness on challenging classes, increasing [email protected] from 0.619 (best single model) to 0.634 on the reviewed dataset. By combining accurate visual detection with knowledge-grounded language generation, the proposed framework provides an interpretable, user-friendly decision support tool for farmers. This work demonstrates the potential of integrating object detection, RAG, and LLMs to support sustainable, data-driven disease management and reduce unnecessary pesticide usage in precision agriculture.