Iso-Memory Comparison of Weight Bit-Widths in Small Quantization-Aware Transformers

At a fixed weight-storage budget, which bits per weight b gives the lowest loss? I cross 7 bit-widths with budgets of 256 Kbit, 828 Kbit and 4 Mbit, resize the model to fill each, and train quantization-aware, 5 seeds per cell, on TinyStories and, at 256 Kbit and 4 Mbit, FineWeb-Edu. Ternary and 2-bit weights lose at every budget, and bf16 where it fits, on every metric computed (their learning rate was not tuned). On preregistered val_loss_best the lowest mean is b=6 at every budget, decisive by my criteria only at 256 Kbit, where precision, depth and MLP ratio covary. At 828 Kbit the smallest gap is within the largest seed sd, so I name no optimum. At 4 Mbit the 0.0061 gap is below the power threshold, and the lowest mean is b=4 on averaged weights and on FineWeb-Edu, so it depends on averaging, on the pilot reading and on the text. Code, configuration, results and OpenTimestamps proofs: https://github.com/milar111/iso-memory-scaling. The 74-page technical report with all derivations and exploratory analyses is attached as technical-report.pdf