How Small Can Sub-1-Bit Go? Lower Limits of LittleBit Binary Low-Rank Compression for Microcontroller-Scale Language Models
LittleBit and LittleBit-2 compress the weights of large language models below one bit per weight (BPW) by factorizing each matrix into binary low-rank factors with floating-point scales. They were demonstrated on models with a billion parameters and more, and motivated by deployment on constrained devices. We ask how far down in model size the method remains useful, and whether it pays off on a real microcontroller. In ten preregistered stages on decoder-only Transformers with 0.32 to 16.9 million parameters, five seeds per configuration and about 1,400 trained models, we find four limits. Structurally, the fixed cost of the floating-point scales keeps whole Transformer blocks above 1 BPW below width 76. In quality, a fully compressed student stays within 1.5 times the perplexity of its uncompressed teacher only from width 256. Against baselines, the student's advantage depends mainly on the baseline: at about 1 BPW it has 22 to 36% lower perplexity than an untied dense FP16 model of equal total memory and, from width 96, 17 to 24% lower than a tied FP16 model, but loses to a tied dense model quantized to 4 bits (INT4) at every width from 32 to 384, by 7.5 to 84% in perplexity; the gap shrinks monotonically above width 64 and is inconclusive at width 512 (ratio 1.033). The same pattern holds on FineWeb-Edu. On hardware, on an ESP32-S3 the deployed models reproduce PyTorch to within about 1e-6 nats, and LittleBit students are 1.5 to 3.2 times slower per token than equal-memory INT4 models with plain C kernels. A one-group-at-a-time ablation shows that compressing the output layer alone to the LittleBit floor roughly doubles perplexity. At this scale, dense INT4 is the better choice for a microcontroller today; an exploratory extrapolation places the quality crossover near width 750. Files: the preprint (20 pages) and its annexes (23 pages), which report every validity check and every confirmatory hypothesis at the level of single cells, the descriptive tables of every stage, the full deviation and incident log and the reproducibility file map.
Authors
- Vladin-Antonio Foca (ORCID: https://orcid.org/0009-0001-0455-9173)
Institutions
- Nicolae Bălcescu Land Forces Academy (RO)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24353773
- Primary Topic
- Advanced Neural Network Applications
- Type
- preprint