Modus_X: Constant-State Language Modeling with Associative Memory and Selective Recurrence

We introduce Modus_X, a novel attention-free causal sequence architecture that integrates two complementary sequence modeling paradigms: selective state-space models (SSMs) for capturing fast, local sequential dynamics, and a content-addressed associative matrix memory using delta-rule updates for long-range associative recall. Unlike traditional Transformers, Modus_X does not employ attention mechanisms or key-value (KV) caches, resulting in O(L) training complexity and O(1) constant inference memory footprint with respect to sequence length. To evaluate the architecture under controlled conditions, we train Modus_X, Transformer, and Mamba-family baselines on the FineWeb-Edu dataset. Our experiments demonstrate that Modus_X achieves superior perplexity compared to the audited Mamba recurrent baseline, while an internal parameter-matched Mamba control suggests that the gain is architectural rather than merely parameter-count driven. Against the Transformer, we use the 40k checkpoint as a reference baseline rather than a strict compute-matched comparison: the Transformer remains lower at 40k, while Modus_X continues to improve through 80k and narrows the short-context language-modeling gap while preserving constant recurrent state. Crucially, synthetic associative recall stress tests indicate the intended qualitative separation: vector-state recurrence suffers interference as the number of independent key-value bindings grows, while the Modus_X matrix stream is designed for content-addressed lookup and overwrite. Modus_X is therefore not yet a Transformer replacement on every metric, but it is a strong attention-free, constant-state contender and a promising second path for long-context modeling.