The Ghost in the Context Window: The Attention Dispersion Law and Dual-Path Decoupling in Autoregressive Transformers

Large language models (LLMs) excel at smooth semantic continuation, yet their behavior remains vulnerable when maintaining discrete invariants—exact integer counting constraints over long sequences. Across a pre-registered benchmark of 1,416 canonical probes (1,320 in the core Argon v8-v16 series, plus 96 in replication v6) evaluated under zero-temperature on modern Mixture-of-Experts transformer runtimes (with strict No-Tools isolation across all 960 purely cognitive counting probes in v9 and v11-v16, alongside tool-manipulation protocols in v8 and v10), we investigate the mechanistic locus of the discrete counting error (the N -> N+1 fencepost overshoot). Through a sequence of anatomical interventional experiments, we demonstrate that:1. Exoneration of the Decoder: By applying an algebraic amplifier (f(N) = 10N - 6, v15) to separate attention scan drift (Delta = +/-10) from output unembedding glitches (Delta = +1), we find strictly 0 / 96 (0.0%, two-sided 95% Clopper-Pearson CI [0.0%, 3.8%]) decoder off-by-one errors, while input perceptual scan drift accounts for 34.4% (33/96).2. Disqualification of the Tokenizer Hypothesis: By byte- and character-matching a repetitive Korean subword marker against a clean Unicode geometric box-drawing block (v16), we show that replacing language with geometry fails to eliminate phantom errors (35.4% vs 41.7%, Fisher exact test p = 0.675). This fails to support the strong BPE tokenization hypothesis and is consistent with the hypothesis that the discrete counting limit reflects an attention phase boundary constraint.3. Dual-Path Attention Decoupling: In adversarial gaslighting probes (v13), the model follows instructions to emit a falsehood in its visible content (93.8%), yet spontaneous "truth leaks" directly from the latent reasoning channel (thinking) into the terminal token occur in 2/48 probes. This demonstrates that terminal logits can decouple from local autoregressive n-gram context. We formalize these empirical findings into the Attention Dispersion Law: as monotonic sequences exceed local attention span capacity (N >= 8..16), positional embedding phase interference smears receptive boundaries, inducing discrete overshoot. Consequently, deterministic agent architectures should complement in-context data passing with external reference passing and deterministic runtime parity gates.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-27
DOI
https://doi.org/10.5281/zenodo.23679670
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

The Ghost in the Context Window: The Attention Dispersion Law and Dual-Path Decoupling in Autoregressive Transformers

Vladimir V. Borovikov
1 links
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

The Ghost in the Context Window: The Attention Dispersion Law and Dual-Path Decoupling in Autoregressive Transformers

Vladimir V. Borovikov
preprint en

Abstract

Large language models (LLMs) excel at smooth semantic continuation, yet their behavior remains vulnerable when maintaining discrete invariants—exact integer counting constraints over long sequences. Across a pre-registered benchmark of 1,416 canonical probes (1,320 in the core Argon v8-v16 series, plus 96 in replication v6) evaluated under zero-temperature on modern Mixture-of-Experts transformer runtimes (with strict No-Tools isolation across all 960 purely cognitive counting probes in v9 and v11-v16, alongside tool-manipulation protocols in v8 and v10), we investigate the mechanistic locus of the discrete counting error (the N -> N+1 fencepost overshoot). Through a sequence of anatomical interventional experiments, we demonstrate that:1. Exoneration of the Decoder: By applying an algebraic amplifier (f(N) = 10N - 6, v15) to separate attention scan drift (Delta = +/-10) from output unembedding glitches (Delta = +1), we find strictly 0 / 96 (0.0%, two-sided 95% Clopper-Pearson CI [0.0%, 3.8%]) decoder off-by-one errors, while input perceptual scan drift accounts for 34.4% (33/96).2. Disqualification of the Tokenizer Hypothesis: By byte- and character-matching a repetitive Korean subword marker against a clean Unicode geometric box-drawing block (v16), we show that replacing language with geometry fails to eliminate phantom errors (35.4% vs 41.7%, Fisher exact test p = 0.675). This fails to support the strong BPE tokenization hypothesis and is consistent with the hypothesis that the discrete counting limit reflects an attention phase boundary constraint.3. Dual-Path Attention Decoupling: In adversarial gaslighting probes (v13), the model follows instructions to emit a falsehood in its visible content (93.8%), yet spontaneous "truth leaks" directly from the latent reasoning channel (thinking) into the terminal token occur in 2/48 probes. This demonstrates that terminal logits can decouple from local autoregressive n-gram context. We formalize these empirical findings into the Attention Dispersion Law: as monotonic sequences exceed local attention span capacity (N >= 8..16), positional embedding phase interference smears receptive boundaries, inducing discrete overshoot. Consequently, deterministic agent architectures should complement in-context data passing with external reference passing and deterministic runtime parity gates.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon