Automatic Assessment of UML Models with Neural Networks: A Case Study for Software Engineering Education

This article investigates the use of neural network-based UML diagram classification to support automated assessment in software engineering education; this may be useful to both teachers and students. For this purpose, our software automatically classifies UML models that conform to the standard Unified Modeling Language (UML), using neural networks trained on an independent dataset composed of public UML repositories and collected student models; thus, starting from models provided by students as images, it distinguishes between various types of UML diagrams, and it also detects whether they do not conform to the standard language. The case study includes a detailed analysis performed for three cohorts totaling 237 students to identify correct assessments, recurring errors, and the impact of diagram quality, missing elements, duplicates, and non-standard notations. To perform an evaluation under imbalanced data conditions, macro average metrics were applied, including precision, recall, specificity, and F1 score. Across the complete evaluation dataset of 881 student submissions, the assessment software achieved a macro average F1 score of 0.96 across the four evaluable ground-truth classes (class, sequence, state, and other UML); the non-UML class was excluded from this macro average because the classroom dataset contained no ground-truth non-UML submissions. These findings demonstrate the feasibility and the limitations of AI-based assessment of students’ UML modeling practical work and highlight opportunities for future integration into student self-assessment and intelligent tutoring systems.