Evaluating Reasoning in Large Language Models with a Modified Think-a-Number Game: Case Study

Background: Large language models (LLMs) excel at various tasks but often encounter difficulties when extended reasoning requires maintaining a consistent internal state.Identifying the threshold at which these systems fail under increasing task complexity is essential for reliable deployment.Objective: The primary objective was to examine whether four LLMs (GPT 3.5, GPT 4, GPT 4o-mini and GPT 4o) could preserve a hidden number and its arithmetic transformation across multiple yes/no queries and to determine whether a specific point of reasoning breakdown exists.Methods: A modified "Think a Number" game was employed, with complexity defined by the number of sequential yes/no queries (ranging from 1 to 9 or 11).Seven prompting strategies, including chain-of-thought variants, counterfactual prompts and few-shot examples, were evaluated.Each outcome was considered correct if the revealed number and transformation of the model remained consistent with prior answers.Results: Analysis of tens of thousands of trials showed no distinct performance cliff up to 9-11 queries, indicating that modern LLMs are more capable of consecutive reasoning than previously assumed.Counterfactual and certain chain-of-thought prompts outperformed simpler baselines.GPT 4o and GPT 4o-mini attained higher overall correctness, whereas GPT 3.5 and GPT 4 more often displayed contradictory or premature disclosures. Conclusion:In a controlled, scalable reasoning scenario, these LLMs demonstrated notable resilience to multi-step prompts.Both prompt design and model selection significantly influenced performance.Further research involving more intricate tasks and higher query counts is recommended to delineate the upper boundaries of LLM internal consistency.