The Model Knows Better Than It Answers: Gemma 3 Composes a Spatial Map Its Answers Never Read

When people read "A is 3 units north of B; C is 2 units east of A", they can answer questions the text never answers, as if looking at a picture. We ask whether a language model builds such a picture, and whether its answers use it. On text-only layouts, Gemma 3 12B builds a map: linear probes read each object's position from its activations (R² = 0.74), including positions that require combining sentences. But the answers do not use this map. Erasing it everywhere costs no more accuracy than erasing random directions. Instead, the answers depend on a small 2-D code over the sentence that states each position. Swapping it moves 72% of the affected answers (83% in Gemma 3 27B; random: 0%), and it acts as a switch, not a coordinate. The answers also use the chain of relations only partly, so answering from the model's own unused map beats the model on chain questions (0.80 vs 0.73). The main tests were preregistered.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-12
DOI
https://doi.org/10.5281/zenodo.24178042
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Model Knows Better Than It Answers: Gemma 3 Composes a Spatial Map Its Answers Never Read

Cristhian Camilo Gomez Neira
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

The Model Knows Better Than It Answers: Gemma 3 Composes a Spatial Map Its Answers Never Read

Cristhian Camilo Gomez Neira
preprint en

Abstract

When people read "A is 3 units north of B; C is 2 units east of A", they can answer questions the text never answers, as if looking at a picture. We ask whether a language model builds such a picture, and whether its answers use it. On text-only layouts, Gemma 3 12B builds a map: linear probes read each object's position from its activations (R² = 0.74), including positions that require combining sentences. But the answers do not use this map. Erasing it everywhere costs no more accuracy than erasing random directions. Instead, the answers depend on a small 2-D code over the sentence that states each position. Swapping it moves 72% of the affected answers (83% in Gemma 3 27B; random: 0%), and it acts as a switch, not a coordinate. The answers also use the chain of relations only partly, so answering from the model's own unused map beats the model on chain questions (0.80 vs 0.73). The main tests were preregistered.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon