How token influence decays with causal distance: Evidence across autoregressive language models from 14M to 32B parameters

Abstract How the sensitivity of autoregressive language models to earlier input tokens depends on causal distance remains incompletely understood. Motivated by local-global decompositions in numerical solvers for differential equations, we study the local input-output response of trained autoregressive Transformers. For target position $j$ and causal distance $r$, we take the Euclidean norm of the gradient of a selected next-token logit with respect to the input embedding at $j-r$. We aggregate profiles by their median. We consider trained autoregressive transformer language models from the Pythia, Qwen2.5, GPT-Neo, Llama~3.1, Gemma~3, SmolLM2, and Mistral families, spanning 14 million to 32 billion parameters, on Gutenberg text. We assess robustness through dataset, token-order, positional, aggregation, and logit-selection controls, complemented by initialization and architectural comparisons. Over the measured distance range, the median sensitivity profiles of trained Transformers decay slowly and are compactly described by a power-law-with-floor model, $$\overline G(r)\approx\gamma+\beta(r+1)^{-p},$$ with fitted exponents ranging from approximately $0.57$ to $0.9$, mostly between $0.65$ and $0.9$. A broken power law sometimes improves the fit, but its breakpoint varies across models and its optimization yields multiple distinct solver endpoints. The profiles are similar for true-token and predicted-token logits. The slow decay is also observed on WikiText-103 and persists under token and sentence shuffling despite degraded prediction quality. It persists with randomly initialized input embeddings, but becomes small and nearly flat when the Transformer backbone and output head are randomized. Trained Pythia-shaped Elman RNNs show rapid sensitivity decay, while LSTM and GRU profiles decay more slowly but then bend downward. The measured quantity is a distance-resolved, aggregated first-order Jacobian response characterizing how sensitivity to localized input perturbations varies with causal source-target distance. These results identify a learned, slowly decaying long-range response that may motivate hierarchical mechanisms combining local interactions with a coarse representation of global coupling.