Leveraging AI for Educational Content Consistency: An Evaluation of Text-to-Text and Image-to-Text Metadata Generation for Online Course Development

We evaluated the performance of two AI tools used for generating metadata documentation for self-paced online learning courses with multiple instructors as a part of our efforts to create more consistent learning experiences. Our objective was to determine whether AI tools could improve searchability and encourage instructors to use our skills taxonomy and media asset library more consistently. The first tool is a custom "Skills Definer GPT" backed by OpenAI’s proprietary GPT-4 model, which was used to generate definitions for skills that we store in a taxonomy database. The second tool is a Python script backed by an open-source Hugging Face model, which was used to generate descriptions for graphical icons that we store in a media library. The purpose of both the taxonomy database and the media library is to maintain metadata that informs instructors how specific skills and icons should be used across multiple lessons and courses within a learning path. Prior to the application of these AI tools, all metadata stored in these systems was written manually, and did not necessarily include sufficient information to ensure that the data was searchable and consistently applied. Then, AI tools were utilized to augment the existing metadata. We evaluated 86 text-to-text skill definitions and 263 image-to-text icon descriptions and achieved moderate inter-annotator agreement. Our analysis found that the majority of the AI-generated metadata did not contain mistakes and could be used for our intended purpose. The mistake rate was considerably higher for the icon descriptions generated by the open-source image-to-text model than for the skill definitions generated by the proprietary text-to-text model. Therefore, while searchability of the metadata improved, manual review would still be required prior to broader application of these definitions and descriptions. Our findings can be utilized by instructors and curriculum designers aiming to improve the consistency of educational content using AI-augmented metadata.