How Small Can Sub-1-Bit Go? Lower Limits of LittleBit Binary Low-Rank Compression for Microcontroller-Scale Language Models

LittleBit and LittleBit-2 compress the weights of large language models below one bit per weight (BPW) by factorizing each matrix into binary low-rank factors with floating-point scales. They were demonstrated on models with a billion parameters and more, and motivated by deployment on constrained devices. We ask how far down in model size the method remains useful, and whether it pays off on a real microcontroller. In ten preregistered stages on decoder-only Transformers with 0.32 to 16.9 million parameters, five seeds per configuration and about 1,400 trained models, we find four limits. Structurally, the fixed cost of the floating-point scales keeps whole Transformer blocks above 1 BPW below width 76. In quality, a fully compressed student stays within 1.5 times the perplexity of its uncompressed teacher only from width 256. Against baselines, the student's advantage depends mainly on the baseline: at about 1 BPW it has 22 to 36% lower perplexity than an untied dense FP16 model of equal total memory and, from width 96, 17 to 24% lower than a tied FP16 model, but loses to a tied dense model quantized to 4 bits (INT4) at every width from 32 to 384, by 7.5 to 84% in perplexity; the gap shrinks monotonically above width 64 and is inconclusive at width 512 (ratio 1.033). The same pattern holds on FineWeb-Edu. On hardware, on an ESP32-S3 the deployed models reproduce PyTorch to within about 1e-6 nats, and LittleBit students are 1.5 to 3.2 times slower per token than equal-memory INT4 models with plain C kernels. A one-group-at-a-time ablation shows that compressing the output layer alone to the LittleBit floor roughly doubles perplexity. At this scale, dense INT4 is the better choice for a microcontroller today; an exploratory extrapolation places the quality crossover near width 750. Files: the preprint (20 pages) and its annexes (23 pages), which report every validity check and every confirmatory hypothesis at the level of single cells, the descriptive tables of every stage, the full deviation and incident log and the reproducibility file map.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24353773
Primary Topic
Advanced Neural Network Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

How Small Can Sub-1-Bit Go? Lower Limits of LittleBit Binary Low-Rank Compression for Microcontroller-Scale Language Models

Vladin-Antonio Foca
Zenodo (CERN European Organization for Nuclear Research)
Advanced Neural Network Applications
preprint

How Small Can Sub-1-Bit Go? Lower Limits of LittleBit Binary Low-Rank Compression for Microcontroller-Scale Language Models

Vladin-Antonio Foca
preprint en

Abstract

LittleBit and LittleBit-2 compress the weights of large language models below one bit per weight (BPW) by factorizing each matrix into binary low-rank factors with floating-point scales. They were demonstrated on models with a billion parameters and more, and motivated by deployment on constrained devices. We ask how far down in model size the method remains useful, and whether it pays off on a real microcontroller. In ten preregistered stages on decoder-only Transformers with 0.32 to 16.9 million parameters, five seeds per configuration and about 1,400 trained models, we find four limits. Structurally, the fixed cost of the floating-point scales keeps whole Transformer blocks above 1 BPW below width 76. In quality, a fully compressed student stays within 1.5 times the perplexity of its uncompressed teacher only from width 256. Against baselines, the student's advantage depends mainly on the baseline: at about 1 BPW it has 22 to 36% lower perplexity than an untied dense FP16 model of equal total memory and, from width 96, 17 to 24% lower than a tied FP16 model, but loses to a tied dense model quantized to 4 bits (INT4) at every width from 32 to 384, by 7.5 to 84% in perplexity; the gap shrinks monotonically above width 64 and is inconclusive at width 512 (ratio 1.033). The same pattern holds on FineWeb-Edu. On hardware, on an ESP32-S3 the deployed models reproduce PyTorch to within about 1e-6 nats, and LittleBit students are 1.5 to 3.2 times slower per token than equal-memory INT4 models with plain C kernels. A one-group-at-a-time ablation shows that compressing the output layer alone to the LittleBit floor roughly doubles perplexity. At this scale, dense INT4 is the better choice for a microcontroller today; an exploratory extrapolation places the quality crossover near width 750. Files: the preprint (20 pages) and its annexes (23 pages), which report every validity check and every confirmatory hypothesis at the level of single cells, the descriptive tables of every stage, the full deviation and incident log and the reproducibility file map.

Zenodo (CERN European Organization for Nuclear Research)
Nicolae Bălcescu Land Forces Academy (RO)
Advanced Neural Network Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon