Citation Gating in Legal RAG: Cost, Consistency and Model Agreement of Proposition-Support Checks for Slovenian Law

Legal research assistants built on large language models can cite real court decisions for propositions those decisions contradict; such a citation is harder to detect than a fabricated one, because it survives a lookup. We study the check against this failure as a deployed pipeline stage for Slovenian law: a citation gate classifies each (proposition, court headnote) pair as supports, complicates, undermines or irrelevant before a citation placed at a claim marker is shown; decisions the analysis model names in its own prose bypass it. The experiments measure system behaviour and agreement with model-derived references, not legal accuracy. On 49 claims from five questions, a five-candidate cap removes 26 of the 36 contrary-authority citations the gate finds at K=20, and cap, gate labels and the rendering limit together change most of what users read: only 28 citations appear in both the 89- and the 99-citation report. A classification-only gate checks every candidate for about USD 0.07 per analysis after GPT-5.4 analyses and returns identical labels on repeat, but its labels change on 28% of pairs when the request is restructured. Against a six-model panel on 368 disputed pairs, drawn largely from its own flags, it flags every contrary authority the other voters agree on (20 of 22 after the production confidence threshold) while they confirm 7% to 11% of its flags. Two of six analysis models named decisions in prose despite instructions; all 49 reached users unchecked. In a preliminary check, one lawyer labelled 50 pairs from the headnote and from the full decision: the labels agreed on 31 of 49 pairs, and two of seven headnote-level undermines labels survived full reading. The product's default configuration was not measured. The 417-pair labelling protocol, code and prompts are released, together with the completed expert labels for the 50 pairs.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-29
DOI
https://doi.org/10.5281/zenodo.23772736
Primary Topic
Artificial Intelligence in Law
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Citation Gating in Legal RAG: Cost, Consistency and Model Agreement of Proposition-Support Checks for Slovenian Law

Nejc Fošnarič
1 links
Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Law
preprint

Citation Gating in Legal RAG: Cost, Consistency and Model Agreement of Proposition-Support Checks for Slovenian Law

Nejc Fošnarič
preprint en

Abstract

Legal research assistants built on large language models can cite real court decisions for propositions those decisions contradict; such a citation is harder to detect than a fabricated one, because it survives a lookup. We study the check against this failure as a deployed pipeline stage for Slovenian law: a citation gate classifies each (proposition, court headnote) pair as supports, complicates, undermines or irrelevant before a citation placed at a claim marker is shown; decisions the analysis model names in its own prose bypass it. The experiments measure system behaviour and agreement with model-derived references, not legal accuracy. On 49 claims from five questions, a five-candidate cap removes 26 of the 36 contrary-authority citations the gate finds at K=20, and cap, gate labels and the rendering limit together change most of what users read: only 28 citations appear in both the 89- and the 99-citation report. A classification-only gate checks every candidate for about USD 0.07 per analysis after GPT-5.4 analyses and returns identical labels on repeat, but its labels change on 28% of pairs when the request is restructured. Against a six-model panel on 368 disputed pairs, drawn largely from its own flags, it flags every contrary authority the other voters agree on (20 of 22 after the production confidence threshold) while they confirm 7% to 11% of its flags. Two of six analysis models named decisions in prose despite instructions; all 49 reached users unchecked. In a preliminary check, one lawyer labelled 50 pairs from the headnote and from the full decision: the labels agreed on 31 of 49 pairs, and two of seven headnote-level undermines labels survived full reading. The product's default configuration was not measured. The 417-pair labelling protocol, code and prompts are released, together with the completed expert labels for the 50 pairs.

Zenodo (CERN European Organization for Nuclear Research)
Peace, justice, and strong institutions
Artificial Intelligence in Law
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Citation Gating in Legal RAG: Cost, Consistency and Model Agreement of Proposition-Support Checks for Slovenian Law — Nejc Fošnarič · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS