Generating UML class diagrams as conceptual models through large language models

Abstract The breakthrough of Large Language Models (LLMs) has changed how different kinds of complex tasks are approached, including the ones that require a higher level of abstraction and critical thinking together with advanced domain-specific knowledge, like Conceptual Modeling. Several experiments on testing the modeling capabilities of LLMs have already been conducted, but the literature still lacks a structured analysis of how different LLMs and prompting techniques impact the extraction of conceptual models, as UML class diagrams, from textual specifications. In this paper, we present a comprehensive comparison of open-source and closed-source LLMs used in conjunction with the most effective and accessible prompting techniques, on a newly crafted high-quality dataset of case specifications, implementing an automated evaluation on generated UML class diagrams. Finally, we assess how factors like model size or case complexity impact the quality of the generated models and what LLM and what prompting technique to choose for which task. The dataset and the experimental source code are made available through GitHub( https://github.com/IlKaiser/text2uml ).