Citation Gating in Legal RAG: Cost, Consistency and Model Agreement of Proposition-Support Checks for Slovenian Law
Legal research assistants built on large language models can cite real court decisions for propositions those decisions contradict; such a citation is harder to detect than a fabricated one, because it survives a lookup. We study the check against this failure as a deployed pipeline stage for Slovenian law: a citation gate classifies each (proposition, court headnote) pair as supports, complicates, undermines or irrelevant before a citation placed at a claim marker is shown; decisions the analysis model names in its own prose bypass it. The experiments measure system behaviour and agreement with model-derived references, not legal accuracy. On 49 claims from five questions, a five-candidate cap removes 26 of the 36 contrary-authority citations the gate finds at K=20, and cap, gate labels and the rendering limit together change most of what users read: only 28 citations appear in both the 89- and the 99-citation report. A classification-only gate checks every candidate for about USD 0.07 per analysis after GPT-5.4 analyses and returns identical labels on repeat, but its labels change on 28% of pairs when the request is restructured. Against a six-model panel on 368 disputed pairs, drawn largely from its own flags, it flags every contrary authority the other voters agree on (20 of 22 after the production confidence threshold) while they confirm 7% to 11% of its flags. Two of six analysis models named decisions in prose despite instructions; all 49 reached users unchecked. In a preliminary check, one lawyer labelled 50 pairs from the headnote and from the full decision: the labels agreed on 31 of 49 pairs, and two of seven headnote-level undermines labels survived full reading. The product's default configuration was not measured. The 417-pair labelling protocol, code and prompts are released, together with the completed expert labels for the 50 pairs.
Authors
- Nejc Fošnarič (ORCID: https://orcid.org/0009-0009-0284-637X)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-29
- DOI
- https://doi.org/10.5281/zenodo.23772736
- Primary Topic
- Artificial Intelligence in Law
- Type
- preprint