Where Do Language Models Hide the Right Answer?
Abstract Students increasingly practise with multiple-choice questions (MCQs) generated by large language models (LLMs). However, an MCQ-based assessment can reliably measure knowledge only when the correct answer is equally likely to occur in every answer position. A recent preprint demonstrated that LLMs exhibit non-uniform answer-position distributions under deterministic decoding. This study investigates a narrower and empirically testable question: whether answer-position bias generalises across model families and versions beyond those examined previously, whether it persists under sampling and prompting conditions representative of routine usage, and whether it can be mitigated through explicit instructions to balance answer keys. We generated 2,819 MCQs, including 2,109 produced using a standard prompt, with gemini-3-flash-preview, gpt-oss-120b, and qwen3.8-27b. The experimental variables included temperature (0–1), batch size (5–25 questions per request), number of answer options (four or five), and prompt wording. Statistical analyses treated the 293 generation requests, rather than individual questions, as the independent experimental units. All three models systematically under-utilised the final answer position, with a pooled empirical frequency of 6.1% (128/2,109), substantially below the expected uniform probability of 20–25%. The preferred answer position remained relatively stable across versions within model families (B for Qwen and B/C for Gemini), whereas the newly examined gpt-oss family exhibited a distinct preference for position A. Variations in temperature, batch size, option count, and prompt wording did not eliminate this bias. An explicit one-sentence instruction to balance answer positions substantially mitigated the bias in two models but was ineffective for Qwen. Moreover, for gpt-oss, this intervention introduced incorrectly keyed questions, affecting 7.2% of the generated items. We further demonstrate the practical implications of this systematic bias: a position-based guessing strategy, calibrated using independent generation requests, correctly predicts 38–57% of previously unseen answers, compared with the nominal chance-level performance of 25%. Finally, we explicitly distinguish the findings that extend previous observations, those that independently reproduce earlier results, and those that qualify existing conclusions. These results highlight the need for systematic answer-key validation and position-bias auditing when LLM-generated MCQs are used for educational assessment.
Authors
- Mayank Goswami (ORCID: https://orcid.org/0000-0001-8829-9980)
- Bhavishay Sheoran (ORCID: https://orcid.org/0009-0000-5298-5217)
Institutions
- Indian Institute of Technology Roorkee (IN)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-17
- DOI
- https://doi.org/10.21203/rs.3.rs-11739168/v1
- Primary Topic
- Topic Modeling
- Type
- preprint