Comparación del desempeño de LLMs en la generación de diagramas de clases UML mediante un sistema RAG

This study presents a comparative evaluation of the performance of six LLMs (Gemini-2.5-Pro, Claude-Sonnet-4, GPT-4, DeepSeek-R1, Llama3.2, and Mistral) within a RAG system for generating UML class diagrams at the analysis level, based on user stories written in natural language. PlantUML code was used for the generation and evaluation of the diagrams, enabling a comparison between the generated diagrams and reference diagrams using the ROUGE-L metric, which focuses on average recall. The results showed that Gemini-2.5-Pro, Claude-Sonnet-4, and GPT-4 achieved better performance, with Claude-Sonnet-4 standing out by obtaining the highest average scores in most user stories. In contrast, DeepSeek-R1, Llama3.2, and Mistral presented difficulties, including the generation of invalid PlantUML code, which limited automated evaluation in some cases. The incorporation of the RAG system provided an initial foundation for exploring improvements in response quality, suggesting limitations in the quality and relevance of the retrieved context. Finally, opportunities for improvement were identified, such as prompt refinement and enhancement of the context used by the RAG system.