How to Automate Feedback on Diagrammatic Reasoning with a Relevant Degree of Freedom?

This paper considers Café 2.0, an Automated Feedback system designed to support students' diagrammatic reasoning in STEM disciplines. Café 2.0 relies on a predefined error library, metamodels, and rules to correct students' solutions and deliver formative feedback. Implementing such a system requires a balance between constraining the solution syntax to enable AF and leaving freedom to students to reflect on their solution. This paper aims to evaluate whether the level of freedom provided by our AF system sufficiently prepares students for exams. In the exam, they must reason and construct solutions starting with a blank page. This study is conducted in an introductory programming course (CS1), based on two semesters (in 2022 and 2023), where Café 2.0 supports online homework. Findings reveal a discrepancy between students' performance in online homework and their success on exams. While many students feel comfortable with fill-in-the-blank diagrams in their homework, they struggle with the open-ended nature of exam tasks. Our results show that, among the students who succeeded in their online homework in 2023, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$20\%$</tex> were still unable to produce any diagram in the exam. Additionally, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$70\%$</tex> of them could not correctly provide a text description of their solution. To overcome this limitation, this paper proposes an enhanced system that integrates predefined rules with Large Language Models (LLMs). In this framework, LLMs serve as translators. Students can freely create their diagrams and annotate them with their own textual descriptions using a drawing editor. The LLM then maps these representations into a more structured format that aligns with predefined rules. In this way, Café 2.0 can generate accurate feedback. This transformed representation retains the same informational content as the original, differing only in format. This feature will offer students greater flexibility in constructing their solutions while ensuring that feedback remains precise and consistent by limiting the role of LLMs to translation rather than feedback generation.