Evaluating the Accuracy and Usability of an Academic Digital Information Assistant Chatbot Based on Retrieval-Augmented Generation for Academic Learning Grant Services

Information services for student grant programmes require answers that are fast, consistent, and grounded in official documents, while the rules themselves change with every programme cycle. This study evaluates AIDA (Academic Digital Information Assistant), a university academic service chatbot, within the domain of an academic learning grant programme. Following a Design Science Research framework, the evaluation proceeded in two cycles. The first measured answer accuracy through the assessment of three programme administrators across 32 test questions on the dimensions of relevance, correctness, and completeness. The second measured usability through the Chatbot Usability Questionnaire administered to 76 student users. The first cycle yielded 90.6% relevance, 81.3% correctness, 90.6% completeness, and an overall pass rate of 81.3%, with inter-rater agreement between 82% and 85%. All six failing items shared a single root cause, namely timeline information that had not been updated to the current programme period; three of them were nonetheless judged relevant and complete, rendering the error difficult for users to detect. The second cycle produced a usability score of 67.72 (SD 11.93) with a reliability of 0.865, highest on ease of use and lowest on conversational naturalness and error handling. The study concludes that the principal constraint on administrative service chatbots lies not in language understanding but in the governance of content freshness, and proposes a knowledge base update mechanism tied to the programme calendar.