Topology First: Sub-1-Bit LLM Compression by Separating Sign Topology from Metric
How far can a large language model be compressed below one bit per weight without heavy retraining? This paper presents NVG, a quantization method developed by the author and named after his Null-Vector Gravity framework. Instead of rounding every weight, NVG stores each layer as a pattern of ±1 signs plus a few 16-bit scales. It chooses the signs from the layer's own activations, solves the scales in closed form and then refines the signs, all without a gradient step. The storage layout comes from LittleBit (arXiv:2506.13771); NVG fills it differently. With no training, on Qwen3.5-0.8B at 1.0 bit per weight, WikiText-2 perplexity is 656 for NVG against 2,719,570 for LittleBit's data-free initializer in the same container. The 8-bit reference scores 24.27, so neither file is usable without training. After training on 10.5 M tokens against its own 8-bit file on one GPU, Qwen3.8-27B shrinks from 27.38 GB to 4.89 GB, with its layers at 0.55 bits per weight and 1.43 over the whole file. It decodes at 71-76 tok/s on one RTX PRO 6000, and its WikiText-2 perplexity is 29.90 against 8.48 for the 8-bit file. Short factual answers can be right, but longer ones repeat and contain errors, so it is not a working model yet. The paper gives the commands, the data and the limits of the method.
Authors
- Oleg Yuryevich Kirichenko
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-03
- DOI
- https://doi.org/10.5281/zenodo.23925182
- Primary Topic
- Advanced Neural Network Applications
- Type
- preprint