Chatbots on a Mid-Scale Knowledge Base: Using an R2 University as an Example

While LLMs like ChatGPT, Claude, and Gemini have proven effective at handling common, everyday tasks, they often lack the domain-specific expertise necessary for more specialized consultations. This limitation makes it essential to integrate a targeted knowledge base when developing chatbots for specific domains. In this study, we investigated the performance of chatbots utilizing large language models (LLMs) within the context of a mid-scale knowledge base for a regional R2 university in the U.S. Using a web-crawled knowledge base, we created chatbots across multiple platforms, testing them against a set of predefined questions to evaluate correctness and comprehensiveness. Our findings indicate that while ChatGPT models, particularly those augmented with both a knowledge base and real-time search capabilities, provide the most balanced responses on objective questions, hallucinations remain an issue. In the subjective testing questions, no single chatbot emerged as the best performer, with different models excelling in various areas based on the personas and questions. The study suggests that while LLMs are valuable tools in education, careful consideration of their limitations is necessary for improving their application in domain-specific consulting.