Design, Validation, and Risk Assessment of LLM-Based Generative AI Systems Operating in the Legal Sector

Large Language Models (LLMs) are increasingly capable of supporting user decisions and performing various tasks autonomously in a wide range of professional domains, including the legal sector. In this work, we describe the general lifecycle of LLM-based systems and the architecture of a system designed to support legal experts in the process of generating maxims from judgments. We propose three application-specific performance metrics and a general-purpose assessment method suitable for the comparison of different systems in any content generation task. Finally, we discuss the results achieved by the system prototype that, according to a board of magistrates, outperforms human experts in the task of generating maxims from judgments.