WIP: Just-in-Time AI Assisted Formative Feedback for Written, Oral, Team-Based Assessment Tasks: What Worked, What Didn't and Why

This innovative practice WIP paper describes the use of three large language model-based pre-trained AI (LLM-based AI), Inflection's Pi, Azure OpenAI GPT-3.5-Turbo and Azure OpenAI GPT-4 models to provide insightful and timely feedback across written, oral, and team-based assessment tasks in a capstone engineering design course. These LLM-based AI could analyze the content of artefacts produced in formative verbal and written tasks, ensuring that the students include relevant information in their report, and the report requirements such as structure, grammar, style, mechanics etc. are met. The models were also used to analyze the meeting transcripts of student teams, thus allowing the teamwork process and contributions from each member of the student team to be monitored closely. The integration of LLM-based pre-trained AI increased the timeliness and effectiveness of formative feedback on students' design and teamwork processes, thereby fostering a more adaptive and personalized learning experience. Any missteps or misunderstandings on the part of the students regarding the task requirements, as well as any issues arising within team interactions can be promptly communicated to the instructor for immediate resolution. While LLM-based pre-trained AI holds significant promise in transforming feedback practice, it is important to acknowledge that there are still limitations and barriers to practical implementation in the classroom. The feedback produced by LLM-based pre-trained AI lacks nuanced contextual understanding. The integration of LLM-based pre-trained AI into feedback practices holds transformative potential for education. On one hand, the models offer the promise of responsive and personalized learning, and the potential to foster critical thinking and problem-solving skills through interactive and adaptive learning platforms. Conversely, the reliability of the models as a feedback tool and students' receptions to their use remains ambiguous. We conclude the paper by proposing some strategies to overcome these limitations and support academics in applying LLM-based pre-trained AI in feedback practice.