Games with hidden facts and blind LLM judges for evaluating legal-advice systems: protocol and preliminary results

Question lists with one "correct" norm do not measure what a tax adviser has to do: notice unmentioned facts, ask for them, and find all governing norms. I describe a protocol that measures this. Opposing language models (Gemini, DeepSeek, GPT and a memoryless Claude instance) invent fictitious German tax cases with hidden facts S1…Sn. My system plays the adviser; it takes the text of every norm only from a local corpus of official legal sources. Blind judges score each hidden item as found, partly found or not found. Game types include trap, deadlines, two norms, corner, time machine and chain of liability. Across three rounds on different cases the share of hidden items found rose from 73 % (45/62) to 84–85 % and 86 % (91 % when disputed scores are settled by the statutory text). The gain came from the order of work of the adviser, not from the models. Of the lost points, 68 % were facts the adviser did not ask the client about, 26 % were missed norms, and 0 % were misread norms. Seventeen claims made by the models were refuted by the statutory text, each checked against the date of the case. Limits: small samples, model judges, judge noise of 0.5–1 point out of 47. Part of a series; overview: https://doi.org/10.5281/zenodo.23949487. Files: English version (authoritative) and German version (Deutsche Fassung). All game cases are fictitious. Not legal or tax advice.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-04
DOI
https://doi.org/10.5281/zenodo.23949535
Primary Topic
Artificial Intelligence in Law
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Games with hidden facts and blind LLM judges for evaluating legal-advice systems: protocol and preliminary results

Sergei Sevriukov
Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Law
preprint

Games with hidden facts and blind LLM judges for evaluating legal-advice systems: protocol and preliminary results

Sergei Sevriukov
preprint en

Abstract

Question lists with one "correct" norm do not measure what a tax adviser has to do: notice unmentioned facts, ask for them, and find all governing norms. I describe a protocol that measures this. Opposing language models (Gemini, DeepSeek, GPT and a memoryless Claude instance) invent fictitious German tax cases with hidden facts S1…Sn. My system plays the adviser; it takes the text of every norm only from a local corpus of official legal sources. Blind judges score each hidden item as found, partly found or not found. Game types include trap, deadlines, two norms, corner, time machine and chain of liability. Across three rounds on different cases the share of hidden items found rose from 73 % (45/62) to 84–85 % and 86 % (91 % when disputed scores are settled by the statutory text). The gain came from the order of work of the adviser, not from the models. Of the lost points, 68 % were facts the adviser did not ask the client about, 26 % were missed norms, and 0 % were misread norms. Seventeen claims made by the models were refuted by the statutory text, each checked against the date of the case. Limits: small samples, model judges, judge noise of 0.5–1 point out of 47. Part of a series; overview: https://doi.org/10.5281/zenodo.23949487. Files: English version (authoritative) and German version (Deutsche Fassung). All game cases are fictitious. Not legal or tax advice.

Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Law
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon