LegalAI Research in LLM Era: Data, Modeling and Evaluation

Abstract Legal artificial intelligence (LegalAI) refers to the use of artificial intelligence technologies to automate various legal tasks. Recent advances in large-scale language models have significantly enhanced the capabilities of LegalAI, marking a new stage in its development. In this paper, we present a comprehensive survey of how large language models (LLMs) are reshaping the research paradigm of LegalAI. Beyond improving task performance, LLMs now serve as integral components across the perspectives of data, modeling, and evaluation. We propose a role-based schema that categorizes the involvement of LLMs along these perspectives and use it to systematically review existing studies in three major legal tasks, including legal classification, legal retrieval, and legal generation. Besides, we conduct a detailed quantitative comparison of LLM effectiveness across roles and tasks, and our findings reveal that the impact of LLMs is shaped by both their assigned roles and the nature of the legal tasks.