Wayfinder Attention: Attention as Learned Search

Dense causal attention is brute-force search plus readout: each query scores the entire prefix, then aggregates values through a softmax mixture. That search is precise but scales quadratically with sequence length. We ask whether dense attention's neighborhoods can be distilled into a learned token-address index that retrieves a top-K set and delegates softmax to ordinary QKV—and whether dedicated addresses matter when the sparse forward budget is held fixed. We introduce Wayfinder Attention, which separates asymmetric address-based retrieval from standard aggregation. A dense teacher defines which tokens matter; multi-positive InfoNCE over teacher top-32 neighbors learns retrieval order, not the full attention matrix; sparse fine-tuning then adapts the model to top-K=128 readout with trainable addresses. In a controlled proof cell (4-layer needle-in-a-haystack; 300-example holdout), Wayfinder matches dense FlashAttention through T=4096. At T=2048, Wayfinder and dense both reach 100.0% exact-match accuracy, versus 81.3% for key-vector top-K sparse attention and 12.3% for local-window attention (W=64)—same forward K, different indexing geometry. A further diagnostic: teacher-neighborhood Recall@128 can exceed Wayfinder Recall@32 for key-vector routing, yet downstream accuracy still drops, so index recall alone does not predict sparse-attention quality. Retrieval shrinks dense QKV edges by a factor of T/K (64× at T=8192, K=128); at T=16384, a fused forward prototype already reaches 2.2× lower latency and 2.4× less peak VRAM than dense FlashAttention. We contribute a reproducible dense-to-sparse recipe, matched-budget mechanism evidence that decoupled addresses beat key-vector routing, and a systems sketch of where learned-address sparsity can cross over. Official code: https://github.com/Wayland-The-Smith/wayfinder-attention (Apache-2.0). This Zenodo record is the paper preprint (CC BY 4.0).