First Impressions on Supporting Students with a Course-Aligned, AI-powered Virtual Teaching Assistant (TA), Oliver

We show in this paper that an LLM-based tutor with a simple, web-based interface is readily used by students to answer natural language queries without specific training. Through analysis of the log data of nearly 3000 separate interactions, we find that students tend to use the tutor to clarify concepts and explain sticking points in laboratory assignments. Results indicate that the chatbot was used throughout the course at times outside of regular help-seeking hours, indicating that the tutor provides personalized support to students who are in need of help even when it is not available. Finally, through analysis of the log data, we can generate useful insights into student learning throughout the course.