A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences
While progress has been made in legal applications, law reasoning, crucial for fair adjudication, remains unexplored.We propose a transparent law reasoning schema enriched with hierarchical factum probandum, evidence, and implicit experience, enabling public scrutiny and preventing bias.Inspired by this schema, we introduce the challenging task, which takes a textual case description and outputs a hierarchical structure justifying the final decision.We also create the first crowd-sourced dataset for this task, enabling comprehensive evaluation.Simultaneously, we propose an agent framework that employs a comprehensive suite of legal analysis tools to address the challenge task.This benchmark paves the way for transparent and accountable AI-assisted law reasoning in the "Intelligent Court" 1 .
Authors
- Fandong Meng (ORCID: https://orcid.org/0000-0002-8158-2377)
- Fei Zheng (ORCID: https://orcid.org/0000-0002-5763-6507)
- Lie Lin (ORCID: https://orcid.org/0000-0003-1670-4558)
- Jie Zhou (ORCID: https://orcid.org/0000-0001-7701-234X)
- Wenjuan Han
- Jinan Xu (ORCID: https://orcid.org/0000-0003-0170-626X)
- Guoyang Ma
- Jiaxin Shen (ORCID: https://orcid.org/0009-0002-9816-1381)
- Huiqi Hu
Publication Details
- Published
- 2025-01-01
- DOI
- https://doi.org/10.18653/v1/2025.findings-acl.887
- Primary Topic
- Law, Economics, and Judicial Systems
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- China Railway
- National Natural Science Foundation of China