Where Do Language Models Hide the Right Answer?

Abstract Students increasingly practise with multiple-choice questions (MCQs) generated by large language models (LLMs). However, an MCQ-based assessment can reliably measure knowledge only when the correct answer is equally likely to occur in every answer position. A recent preprint demonstrated that LLMs exhibit non-uniform answer-position distributions under deterministic decoding. This study investigates a narrower and empirically testable question: whether answer-position bias generalises across model families and versions beyond those examined previously, whether it persists under sampling and prompting conditions representative of routine usage, and whether it can be mitigated through explicit instructions to balance answer keys. We generated 2,819 MCQs, including 2,109 produced using a standard prompt, with gemini-3-flash-preview, gpt-oss-120b, and qwen3.8-27b. The experimental variables included temperature (0–1), batch size (5–25 questions per request), number of answer options (four or five), and prompt wording. Statistical analyses treated the 293 generation requests, rather than individual questions, as the independent experimental units. All three models systematically under-utilised the final answer position, with a pooled empirical frequency of 6.1% (128/2,109), substantially below the expected uniform probability of 20–25%. The preferred answer position remained relatively stable across versions within model families (B for Qwen and B/C for Gemini), whereas the newly examined gpt-oss family exhibited a distinct preference for position A. Variations in temperature, batch size, option count, and prompt wording did not eliminate this bias. An explicit one-sentence instruction to balance answer positions substantially mitigated the bias in two models but was ineffective for Qwen. Moreover, for gpt-oss, this intervention introduced incorrectly keyed questions, affecting 7.2% of the generated items. We further demonstrate the practical implications of this systematic bias: a position-based guessing strategy, calibrated using independent generation requests, correctly predicts 38–57% of previously unseen answers, compared with the nominal chance-level performance of 25%. Finally, we explicitly distinguish the findings that extend previous observations, those that independently reproduce earlier results, and those that qualify existing conclusions. These results highlight the need for systematic answer-key validation and position-bias auditing when LLM-generated MCQs are used for educational assessment.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-17
DOI
https://doi.org/10.21203/rs.3.rs-11739168/v1
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Where Do Language Models Hide the Right Answer?

Mayank Goswami, Bhavishay Sheoran
2 links
Research Square
Topic Modeling
preprint

Where Do Language Models Hide the Right Answer?

Mayank Goswami, Bhavishay Sheoran
preprint en

Abstract

Abstract Students increasingly practise with multiple-choice questions (MCQs) generated by large language models (LLMs). However, an MCQ-based assessment can reliably measure knowledge only when the correct answer is equally likely to occur in every answer position. A recent preprint demonstrated that LLMs exhibit non-uniform answer-position distributions under deterministic decoding. This study investigates a narrower and empirically testable question: whether answer-position bias generalises across model families and versions beyond those examined previously, whether it persists under sampling and prompting conditions representative of routine usage, and whether it can be mitigated through explicit instructions to balance answer keys. We generated 2,819 MCQs, including 2,109 produced using a standard prompt, with gemini-3-flash-preview, gpt-oss-120b, and qwen3.8-27b. The experimental variables included temperature (0–1), batch size (5–25 questions per request), number of answer options (four or five), and prompt wording. Statistical analyses treated the 293 generation requests, rather than individual questions, as the independent experimental units. All three models systematically under-utilised the final answer position, with a pooled empirical frequency of 6.1% (128/2,109), substantially below the expected uniform probability of 20–25%. The preferred answer position remained relatively stable across versions within model families (B for Qwen and B/C for Gemini), whereas the newly examined gpt-oss family exhibited a distinct preference for position A. Variations in temperature, batch size, option count, and prompt wording did not eliminate this bias. An explicit one-sentence instruction to balance answer positions substantially mitigated the bias in two models but was ineffective for Qwen. Moreover, for gpt-oss, this intervention introduced incorrectly keyed questions, affecting 7.2% of the generated items. We further demonstrate the practical implications of this systematic bias: a position-based guessing strategy, calibrated using independent generation requests, correctly predicts 38–57% of previously unseen answers, compared with the nominal chance-level performance of 25%. Finally, we explicitly distinguish the findings that extend previous observations, those that independently reproduce earlier results, and those that qualify existing conclusions. These results highlight the need for systematic answer-key validation and position-bias auditing when LLM-generated MCQs are used for educational assessment.

Research Square
Indian Institute of Technology Roorkee (IN)
Quality education
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon