Methodology for Evaluating Artificial Intelligence Chatbots: A Review with Applications to Agricultural Advisory Systems
Artificial intelligence chatbots increasingly advise farmers, yet their adoption should rest on rigorous, standardised evaluation. This review brings together five standardised instruments for assessing chatbots (the System Usability Scale, the Chatbot Usability Scale, the Checklist for Trust between People and Automation, the AI Chatbots Usage Scale, and the User Engagement Scale) and one assessment method, knowledge testing. It also explains Cronbach’s alpha, the reliability criterion most of them rely on. A case study from the literature, in which five large language models answered PhD-level soil science questions, together with deployed systems such as Farmer.Chat and AgroLLM, show how evaluation works in practice and why fluent but unverified advice is risky. Beyond questionnaires, we examine accuracy-based and retrieval-grounded testing. We propose a five-layer protocol that first checks technical accuracy, then usability, trust, engagement, and learning outcomes, to guide the design and validation of chatbots for precision and smallholder farming.
Authors
- Farhana L S (ORCID: https://orcid.org/0009-0001-7702-5586)
Institutions
- Kerala Agricultural University (IN)
Publication Details
- Journal
- Journal of Sustainable Technology in Agriculture
- Published
- 2026-11-15
- DOI
- https://doi.org/10.65287/josta.202610.de78
- Primary Topic
- Smart Agriculture and AI
- Type
- article
- Field-Weighted Citation Impact
- 0.00