Supervised Optimism Correction: Be Confident When LLMs Are Sure

In this work, we establish a novel theoretical connection between supervised fine-tuning and offline reinforcement learning under the tokenlevel Markov decision process, revealing that large language models indeed learn an implicit Q-function for inference.Through this theoretical lens, we demonstrate that the widely used beam search method suffers from unacceptable over-optimism, where inference errors are inevitably amplified due to inflated Q-value estimations of suboptimal steps.To address this limitation, we propose Supervised Optimism Correction (SOC), which introduces a simple yet effective auxiliary loss for token-level Qvalue estimations during supervised fine-tuning.Specifically, the auxiliary loss employs implicit value regularization to boost model confidence in expert-demonstrated responses, thereby suppressing over-optimism toward insufficiently supervised responses.Extensive experiments on mathematical reasoning benchmarks, including GSM8K, MATH, and GAOKAO, showcase the superiority of the proposed SOC with beam search across a series of open-source models.