THE ORIGIN OF SELF-ATTENTION: FROM PAIRWISE AFFINITY MATRICES TO TRANSFORMERS

This paper traces the conceptual lineage of self-attention mechanisms across domains—vision, language, and graph learning—by highlighting their common foundation in affinity matrices. It positions Infinite Feature Selection (Inf-FS) as a general precursor to Transformer-style attention, showing that self-attention is a special case of broader affinity-based computation. The work unifies developments in attention, feature selection, graph reasoning, and deep learning architectures, arguing that the core mechanism—reasoning over pairwise relationships—is conserved across domains. This unified view emphasizes that the Transformer’s success stems from a long-standing principle: leveraging pairwise similarities to guide information flow.<br>