Robin: An Alternative to Attention with Block-Parallel State-Space Processing

The dominant paradigm in text generation today relies on the Transformer architecture and its self-attention mechanism. While effective, attention suffers from quadratic complexity with sequence lengthand is constrained by a fixed context window, which often falls short of real-world requirements such asprocessing entire documents or long conversations. Recent research has explored State-Space Models (SSMs) as alternatives, with approaches such as Mamba demonstrating that recurrent state updates can replaceattention for long-context modeling. However, existing SSM-based models are difficult to parallelize duringtraining and are typically not optimized for the GPU-centric hardware ecosystem, which was originallydesigned around parallelizable Transformer workloads. In this work we introduce Robin, a new architecture that combines the sequential memory update principle of SSMs with a block-parallel design tailored for efficient GPU execution. At the core of Robin are lightweight memory blocks that update state vectorstoken by token without relying on positional encodings, while allowing parallel training through block factorization. This design reduces parameter redundancy compared to standard SSMs, preserves long-contextcapabilities, and achieves higher efficiency on commodity hardware. We believe that Robin offers a promisingbalance between scalability, efficiency, and modeling power, positioning it as a practical alternative to attention-based architectures.