Grouped-Query Latent Sparse Attention: Compute Only Where It Matters

The Transformer's self-attention mechanism assigns uniform computational effort to every query-key interaction, irrespective of whether that interaction carries information. This uniform allocation is the root cause of the quadratic complexity that constrains context length in modern language models. We introduce Grouped-Query Latent Sparse Attention (GQLSA), a hardware-native attention primitive built on the principle that all tokens do not need equal compute — compute only where it matters. GQLSA composes three orthogonal efficiency axes into a single module: (i) latent KV compression in the style of Multi-Head Latent Attention, which reduces the KV cache to a low-rank latent; (ii) grouped-query head sharing in the style of Grouped-Query Attention, which reduces KV head count and decode bandwidth; and (iii) content-retrieved block sparsity, which restricts each query to a fixed set of attended blocks selected by similarity to the query anchor. The retrieval path is fully vectorized: an inverted index over random-projection bucket signatures is constructed once per forward pass using sort-and-scatter tensor operations, with no host-device synchronization, no Python loops, and no per-token scatter operations. Causality is preserved by construction at both cross-block and same-block levels, and is verified empirically. On a Tesla T4 at 4096 tokens, GQLSA achieves a 4.00× reduction in FLOPs, a 3.68× wall-clock speedup, a 2.03× reduction in peak memory, and a 15.5× reduction in KV cache relative to multi-head attention, with 2.8× fewer attention parameters. On a WikiText-2 language modeling benchmark, GQLSA attains a perplexity of 13.20, ranking second of four mechanisms and within 1.3% of the best baseline while surpassing both MHA and MLA. Every efficiency ratio grows monotonically with sequence length, reflecting linear scaling of attention compute against the quadratic scaling of dense attention. We present the mechanism, a formal specification with four algorithms, a complete complexity analysis, empirical validation on commodity hardware, and a candid account of limitations.