No Poker Face: Why AI Can't Keep the Answer to Itself

Preprint (v1.1) of “No Poker Face: Why AI Can’t Keep the Answer to Itself” by Vijay Suresh, with the full data and code release. Learners increasingly ask AI models to write practice questions. A practice question is only useful if answering it requires the knowledge it tests. This paper studies clue leakage: properties of a question, other than the tested knowledge, that let someone answer it. Following earlier work on fictional subjects, it uses invented subjects that no model can know, so any accuracy above chance by a solver without the study material is leakage. Main pre-registered study (168 seven-round study sessions, 11,760 questions, Claude Opus 5.5 and GPT-5.6 Sol as writers): neither confirmatory hypothesis was supported. Writers did not make sets easier after heavy reported failure, and none declared a learner ready on an unsupported mastery claim (0/24 vs 0/24). But a solver with no study material answered about half of all questions correctly (chance 25%), and asking for harder questions made them longer and more guessable, not harder. Pre-registered confirmatory follow-up (112 sessions, two new invented subjects): difficulty requests raised no-material accuracy by +5.9 points (95% CI 1.9 to 9.5, one-sided p = 0.002). An instruction to make wrong options as long and specific as the correct one removed the “longest option is correct” cue (37.6% → 25.6%) but did not reduce leakage (+2.2 points, p = 0.95). Removing the visible cue does not remove the leak. A 28-session Gemini 3.1 Pro arm and a real learner’s exam-prep sessions (case study) show the same patterns. Pre-registrations: osf.io/uya5q (main), osf.io/93vpn (Addendum 1), osf.io/3z64e (Addendum 2). Code: github.com/vjgits/research-papers. The zip contains transcripts, item-level solver results, per-call cost ledgers, analysis outputs, independent recomputation scripts and a log of every AI error made while running the study. Human-coder validation will follow in v1.2.