Characterizing Human and LLM Responses in Low-Resource Swahili Agricultural Information Retrieval

Large language models (LLMs) are increasingly used in conversational information retrieval systems to provide direct answers to user queries. However, little is known about how LLM-generated responses behave in low-resource language settings, particularly in domain-specific contexts such as agricultural knowledge access. This study investigates whether answers produced by LLMs differ systematically from human-authored responses in a Swahili agricultural question answering task. Using a dataset of 127 maize-related questions collected through fieldwork in Morogoro, Tanzania, we compare human responses with answers generated by two conversational models, ChatGPT and Gemini. The analysis combines descriptive textual statistics, lexical similarity measures based on TF-IDF cosine similarity, and TF-IDF–based logistic regression evaluated through question-grouped cross-validation. Results show consistent differences in response length, lexical diversity, and lexical similarity, with LLM-generated answers exhibiting longer and more uniform lengths despite instructions to answer briefly. Human answers had higher raw type-token ratios, while model answers showed higher fixed-window lexical diversity. Same-question lexical similarity exceeded a mismatched-question baseline. The three-source classifier achieved 0.9633 accuracy, compared with 0.7192 using length alone and 0.8058 after matching answer lengths. These findings highlight detectable patterns associated with both verbosity and source-specific wording, providing insights for evaluating conversational information retrieval systems in low-resource agricultural environments.