Hello, Freire! How Can You Help Me? A Smart Chatbot to Enhance Access to Academic Opportunities and Institutional Matters

Context: Universities offer various resources and services that students often fail to utilize due to difficulties in accessing relevant information. Chatbots can improve access and interaction between students and institutions. Problem: Students struggle to find information about academic opportunities and institutional matters due to fragmented data, limiting engagement. Solution: This paper introduces Freire Assistant (Freire), a chatbot system powered by a Large Language Model (LLM) and built using Retrieval Augmented Generation (RAG) pipelines. Freire aims to streamline access to academic information through a fast, intuitive communication channel. IS theory: Thiswork was developed under the aegis of Soft Systems Theory, as it addresses the complexity of accessing academic information as a sociotechnical problem. Method: To evaluate Freire’s effectiveness in providing accurate and relevant information about academic opportunities and institutional matters, we conducted a case study at a Brazilian university. We quantitatively evaluated the contextual quality of Freire’s responses using the DeepEval framework. Additionally, we qualitatively evaluated Freire’s usability with 45 students through the Chatbot Usability Questionnaire (CUQ) which measures aspects related to Chatbot Personality, Onboarding, User Experience, and Error Handling. Summary of Results: The findings demonstrated Freire’s efficiency in generating contextually relevant and accurate responses. Furthermore, usability scores were generally high due to their simple interface and conversation-driven functionality. Contributions and Impact in the IS area: This study highlights how smart chatbots can enhance user experience and access to information in universities, providing a replicable model for other institutions facing similar challenges.