Condor: A Neural Connection Network for Enhanced Attention based on Generalized Local Analysis

The attention mechanism in traditional neural networks relies on pairwise interactions between tokens, limiting its ability to capture complex, multi-token relationships. This study introduces Condor, a novel architecture that extends theattention mechanism through a neural connection network based on the GeneralizedLocal Analysis (GLA) framework. Our approach replaces static attention patternswith learnable basis functions that dynamically model relationships within a local window. The Condor architecture achieves linear computational complexity ofO(LWH) while maintaining the expressive power to capture sophisticated sequencepatterns. Experimental results on wikitext-2 demonstrate improved perplexity andfaster convergence compared to the standard Transformer, confirming that each attention head learns unique connection patterns specializing in different aspects of sequence modeling. Code is available at: https://github.com/Kim-Ai-gpu/Condor