The Byte Tax: Decode Speed of Byte-Level Language Models on a Consumer GPU

Byte-level language models remove the tokenizer, but at batch size 1 they decode more slowly per generated byte than the token model they come from. We measure this byte tax for Bolmo 1B, a byteified OLMo 2 1B, against its source on a consumer GPU (RTX 3060 Ti), through a chain of 67 hypotheses written down before each measurement. In eager decoding Bolmo is 2.36× slower per byte. Most of the gap is launch and host overhead in its per-byte local encoder and decoder: CUDA graphs and a single-sync decode loop make Bolmo 3.8× faster, byte-identically, and leave a gap of 1.5× that follows a memory-bandwidth model. Weight quantization confirms the model and shows that the per-byte local stack is the cheapest part to quantize, saving 4.1× more time per gigabyte than the global transformer. Speculative drafting first widens the tax to 2.37×: a token model verifies several drafted tokens almost for free, and a byte model’s drafting rounds end and restart at byte granularity. Aligning drafting rounds to patch boundaries reverses this. It makes Bolmo 2.06× faster than its plain loop in bf16, exact in fp32, and narrows the drafted tax to 1.36×, below the undrafted 1.49×. Optimizations of weight reads favour the token model, and at int8 the tax settles at 1.60×, about 13% above the bandwidth floor for that configuration. A kernel census and microbenchmarks show that the remaining overhead is kernel count, not synchronization, and that a persistent megakernel would gain only 4 to 8%. Overall Bolmo becomes 10.3× faster than eager, and 26 of the 67 hypotheses were refuted. Files in this record byte_tax_paper.pdf: the paper (10 pages). byte_tax_technical_report.pdf: the technical report, the full record of all 13 studies and the 67 pre-registered hypotheses with verdicts, including the negative results. byteprof-1.0.0.zip: snapshot of the code at v1.0.0 (profiling harness, kernels, configurations, scripts, tests; Apache-2.0). Also on GitHub: github.com/fabs133/byteprof. latex_sources.zip: LaTeX sources of the paper and the report. study_records.zip: FINDINGS.md with every hypothesis as written before its measurement, the task specification of each study, run logs and notes. results_core.zip: raw results of every run (reports, JSON summaries, quality evaluations). results_traces.zip: large profiler traces (torch.profiler, Nsight Systems) and the drafting corpus. README_ZENODO.md: guide to the files, hardware, licenses and reproduction. SHA256SUMS.txt: checksums of all files above. Licenses: code under Apache-2.0; paper, report and data under CC BY 4.0. All measurements: RTX 3060 Ti (8 GB), Windows, batch size 1.