Interactive Rubric Generator for Instructor's Assessment Using Prompt Engineering and Large Language Models

This research paper describes an interactive system using large language models and prompt engineering to generate rubrics. Rubrics have long been employed to ensure a grading system that is both equitable and consistent. In practice, generating rubrics could be challenging for instructors for many reasons (e.g., a tight course schedule, limited resources, and varying materials for different projects in the same course), which urges the need to generate the rubric automatically. To the best of our knowledge, little research has been performed on generating rubrics. In this work, we present a novel system based on Large Language Models (LLMs) and Prompt Engineering to help instructors generate rubric items interactively based on course materials, as well as assess the student's work using these rubrics to give timely feedback automatically. In this system, we applied several LLMs (e.g., GPT4, Llama, Falcon, and Hermes) to generate both rubric and feedback using this process: 1) a set of text chunks are initially generated from the textual materials (these textual materials may from various sources), then LDA (Latent Dirichlet Allocation) is applied to extract a set of keywords from the preprocessed text chunks for rubric generation; 2) a web page was designed to let the instructor choose if the keywords from the set are adequate as rubric words; 3) the rubric items are generated by LLMs from the rubric words. In our experiments, a total number of 1017 documents (including the syllabus, the course website, the requirement of projects, the students' works, and the instructors' feedback) were used to build the corpus to generate the rubric-related keywords. Three users (including one instructor and two teaching assistants) participated in generating the rubric interactively using the webpage. The results of experiments show that the interactively generated rubrics from the LLM-instructor system can achieve a level similar to manually created rubrics. We utilized different prompts to let the LLMs generate feedback for the student's work, based on the generated rubrics. Our study shows that generating automatic rubrics and feedback for student project reports is feasible, yet it also identifies significant challenges that future research needs to address.