Methodology for Evaluating Artificial Intelligence Chatbots: A Review with Applications to Agricultural Advisory Systems

Artificial intelligence chatbots increasingly advise farmers, yet their adoption should rest on rigorous, standardised evaluation. This review brings together five standardised instruments for assessing chatbots (the System Usability Scale, the Chatbot Usability Scale, the Checklist for Trust between People and Automation, the AI Chatbots Usage Scale, and the User Engagement Scale) and one assessment method, knowledge testing. It also explains Cronbach’s alpha, the reliability criterion most of them rely on. A case study from the literature, in which five large language models answered PhD-level soil science questions, together with deployed systems such as Farmer.Chat and AgroLLM, show how evaluation works in practice and why fluent but unverified advice is risky. Beyond questionnaires, we examine accuracy-based and retrieval-grounded testing. We propose a five-layer protocol that first checks technical accuracy, then usability, trust, engagement, and learning outcomes, to guide the design and validation of chatbots for precision and smallholder farming.

Authors

Institutions

Publication Details

Journal
Journal of Sustainable Technology in Agriculture
Published
2026-11-15
DOI
https://doi.org/10.65287/josta.202610.de78
Primary Topic
Smart Agriculture and AI
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Methodology for Evaluating Artificial Intelligence Chatbots: A Review with Applications to Agricultural Advisory Systems

Farhana L S
Journal of Sustainable Technology in Agriculture
Smart Agriculture and AI
article

Methodology for Evaluating Artificial Intelligence Chatbots: A Review with Applications to Agricultural Advisory Systems

Farhana L S
article en

Abstract

Artificial intelligence chatbots increasingly advise farmers, yet their adoption should rest on rigorous, standardised evaluation. This review brings together five standardised instruments for assessing chatbots (the System Usability Scale, the Chatbot Usability Scale, the Checklist for Trust between People and Automation, the AI Chatbots Usage Scale, and the User Engagement Scale) and one assessment method, knowledge testing. It also explains Cronbach’s alpha, the reliability criterion most of them rely on. A case study from the literature, in which five large language models answered PhD-level soil science questions, together with deployed systems such as Farmer.Chat and AgroLLM, show how evaluation works in practice and why fluent but unverified advice is risky. Beyond questionnaires, we examine accuracy-based and retrieval-grounded testing. We propose a five-layer protocol that first checks technical accuracy, then usability, trust, engagement, and learning outcomes, to guide the design and validation of chatbots for precision and smallholder farming.

Journal of Sustainable Technology in AgricultureVol. 2(4)
Kerala Agricultural University (IN)
Smart Agriculture and AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon