Do Language Models Prefer Better Demonstrations or Easier Output Tokens?
Model feedback is often used to choose in-context demonstrations, but a high probability for the correct answer token may reflect token compatibility rather than useful example content. We audit this issue with Qwen2.5-0.5B-Instruct on BoolQ using corrected inputs, frozen samples, and query-level inference. Under the original Yes/No labels, the preferred same-label demonstration improves the held-out margin over a frozen random baseline by 0.2526. Late-layer residual interventions, however, show no reliable advantage for a matched donor over an unrelated donor with the same label. Counterbalanced A/B mappings then make the raw preferred-minus-random effect follow the output token. A new frozen extension compares raw correct-token likelihood, a binary logit margin, and content-free calibration. The raw selector is harmful on average and strongly favors B. Binary-margin selection gives the largest average held-out gain but leaves a smaller bias toward A. Content-free calibration reduces the raw token gap but does not meet the preset equivalence rule. These results show that model-based demonstration preference depends on the full prompt, label vocabulary, and scoring rule. It should not be read as a direct measure of semantic example quality.
Authors
- Jiaye Zhang
Institutions
- ShanghaiTech University (CN)
Publication Details
- Journal
- Computers and artificial intelligence.
- Published
- 2026-10-29
- DOI
- https://doi.org/10.70267/cai.v3n5.0110
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 0.00