CM-Galaxy: A Sub-Quadratic, Attention-Free Backbone for Cheap Long-Context Modeling (Teaser v0)

CM-Galaxy is a small (~25M-parameter), attention-free, sub-quadratic (O(n log n)) language-model backbone for cheap long-context modeling, in both training and inference. It combines a long-convolution / FFT-family token mixer with a gated delta-rule memory block, giving a constant-size decode state (no growing KV-cache). This teaser (v0) reports measured results and honest limits: clean associative recall on par with attention (MQAR), long-gap induction recall that holds up better than a standard attention baseline (fair, same-harness comparison to 16k), ~37.5 perplexity and ~18.1% LAMBADA at 8k context, flat decode memory (~312 MB from 8k to 8.4M tokens vs a linearly-growing attention KV-cache), and high-batch throughput advantages. Absolute long-range recall remains modest and is the central open problem; scale and state-space comparisons are untested. The architecture recipe, training details, source code, and checkpoints are not released at this stage. A standalone standard-transformer baseline is provided for independent verification. Repository: https://github.com/singhmandy25-gif/cm-galaxy