Benchmarking latent-space reasoning abilities in large language models through forced response-language selection

Abstract Large language models (LLMs) can perform reasoning computations both internally within their latent space and externally by generating explicit token sequences like chains of thought. Significant progress in enhancing reasoning abilities has been made by scaling test-time compute. However, understanding and quantifying model-internal reasoning abilities – the inferential “leaps” models make between individual token predictions – remains crucial. This study introduces a benchmark (n = 4,000 items) designed to quantify model-internal reasoning in different domains. We achieve this by having LLMs indicate the correct solution to reasoning problems not through descriptive text, but by selecting a specific language of their initial response token that is different from English, the benchmark language. This not only requires models to reason beyond their context window, but at the same time to overcome their default behavior to respond in the same language as the prompt, posing an additional cognitive strain. We evaluate a set of 18 LLMs, showing significant performance variations, with GPT-4.5 achieving the highest accuracy (74.7%), outperforming models like Grok-2 (67.2%), or Llama 3.1 405B (65.6%). Control experiments and difficulty scaling analyses suggest that while LLMs engage in internal reasoning, we cannot rule out heuristic exploitations under certain conditions, marking an area for future investigation. Our experiments demonstrate that LLMs can “think” via latent-space computations, highlighting model-internal inference processes that need further understanding, especially in view of safety-related aspects like covert planning, goal-seeking, or deception emerging without explicit token traces. Significance Statement – Large language models (LLMs) often appear to "reason" only when they show step-by-step chains of thought. We introduce a benchmark that instead measures reasoning carried out silently in a model’s latent space. Tasks require models to infer the correct answer and indicate it solely by choosing a pre-specified non-English response language—preventing them from "thinking in the context window." Across eight reasoning domains, models show wide variation, with some large dense transformer models substantially outperforming others. Control studies and difficulty scaling suggest genuine internal computation, while also revealing potential heuristic shortcuts. By isolating model-internal "reasoning leaps," our approach helps quantify hidden inferences and highlights oversight and safety risks from capabilities that leave no explicit token trace.