Games with hidden facts and blind LLM judges for evaluating legal-advice systems: protocol and preliminary results
Question lists with one "correct" norm do not measure what a tax adviser has to do: notice unmentioned facts, ask for them, and find all governing norms. I describe a protocol that measures this. Opposing language models (Gemini, DeepSeek, GPT and a memoryless Claude instance) invent fictitious German tax cases with hidden facts S1…Sn. My system plays the adviser; it takes the text of every norm only from a local corpus of official legal sources. Blind judges score each hidden item as found, partly found or not found. Game types include trap, deadlines, two norms, corner, time machine and chain of liability. Across three rounds on different cases the share of hidden items found rose from 73 % (45/62) to 84–85 % and 86 % (91 % when disputed scores are settled by the statutory text). The gain came from the order of work of the adviser, not from the models. Of the lost points, 68 % were facts the adviser did not ask the client about, 26 % were missed norms, and 0 % were misread norms. Seventeen claims made by the models were refuted by the statutory text, each checked against the date of the case. Limits: small samples, model judges, judge noise of 0.5–1 point out of 47. Part of a series; overview: https://doi.org/10.5281/zenodo.23949487. Files: English version (authoritative) and German version (Deutsche Fassung). All game cases are fictitious. Not legal or tax advice.
Authors
- Sergei Sevriukov
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-04
- DOI
- https://doi.org/10.5281/zenodo.23949535
- Primary Topic
- Artificial Intelligence in Law
- Type
- preprint