Greedy Byte-Pair Encoding Achieves 0.391 of the Optimum, Past the One-Third Barrier of Pair Packing

Byte-Pair Encoding (BPE) builds most language-model tokenisers by repeatedly merging a most frequent adjacent pair of symbols. How much shortening can this greedy rule lose against the best k merges? Kozma and Voderholzer proved that greedy achieves at least one third of the optimal shortening; the best known upper bound is (1-e^{-2})/2 ≈ 0.432. We show that one third is a barrier for their method, which sees the optimum only through a pair packing; we then prove that greedy achieves at least 0.391 of the optimum on every input, with any tie-breaking. The key idea is rule determinism: all copies of a rule of the optimal merge sequence spell the same letters, so when a greedy replacement destroys three of the optimum's merge points, half a unit is refunded on each side. A weaker bound, 0.374, has a short computer-free proof; the full bound uses a computer-checked certificate. We state one per-step inequality that would give the tight constant, and prove it for the first three steps and for optimal tokens of length at most five. The record includes the paper, its LaTeX source, and the code and certificates that reproduce every computed result.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-06
DOI
https://doi.org/10.5281/zenodo.24023719
Citations
1
Primary Topic
Algorithms and Data Compression
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Greedy Byte-Pair Encoding Achieves 0.391 of the Optimum, Past the One-Third Barrier of Pair Packing

Kundan Baliga, Aditya Byju, Aiden Xie
1 citations
Zenodo (CERN European Organization for Nuclear Research)
Algorithms and Data Compression
preprint

Greedy Byte-Pair Encoding Achieves 0.391 of the Optimum, Past the One-Third Barrier of Pair Packing

Kundan Baliga, Aditya Byju, Aiden Xie
preprint en
1 citations

Abstract

Byte-Pair Encoding (BPE) builds most language-model tokenisers by repeatedly merging a most frequent adjacent pair of symbols. How much shortening can this greedy rule lose against the best k merges? Kozma and Voderholzer proved that greedy achieves at least one third of the optimal shortening; the best known upper bound is (1-e^{-2})/2 ≈ 0.432. We show that one third is a barrier for their method, which sees the optimum only through a pair packing; we then prove that greedy achieves at least 0.391 of the optimum on every input, with any tie-breaking. The key idea is rule determinism: all copies of a rule of the optimal merge sequence spell the same letters, so when a greedy replacement destroys three of the optimum's merge points, half a unit is refunded on each side. A weaker bound, 0.374, has a short computer-free proof; the full bound uses a computer-checked certificate. We state one per-step inequality that would give the tight constant, and prove it for the first three steps and for optimal tokens of length at most five. The record includes the paper, its LaTeX source, and the code and certificates that reproduce every computed result.

Zenodo (CERN European Organization for Nuclear Research)
Georgia Institute of Technology (US), University of Illinois Urbana-Champaign (US)
Algorithms and Data Compression
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon