THE ORIGIN OF SELF-ATTENTION: FROM PAIRWISE AFFINITY MATRICES TO TRANSFORMERS

The self-attention mechanism, now central to deep learning architectures such as Transformers, represents a modern instantiation of a broader computational principle: learning and leveraging pairwise affinity matrices to modulate information flow. This paper traces the conceptual lineage of self-attention across domains—vision, language, and graph learning - through the shared structure of an affinity matrix A. In particular, we spotlight Infinite Feature Selection (Inf-FS) as a foundational framework that generalizes the notion of affinity-based weighting. Unlike the fixed dot-product formulation in Transformers, Inf-FS defines A flexibly—either handcrafted or learned—and computes feature importance through multi-hop propagation over the affinity graph. Methodologically, this positions Inf-FS as a superset: self-attention arises as a specific case where A is parameterized via learned token similarities and applied in a single-hop fashion. We argue that the core structure—reasoning over pairwise relationships—is conserved, and the main distinction lies in how A is constructed and utilized. By reframing self-attention within the broader paradigm of affinity-based computation, we unify disparate threads in machine learning and underscore a shared mathematical foundation that transcends task or architecture.<br>