Greedy byte-pair encoding asymptotically maximizes token savings on fixed alphabets
An explicit binary construction at the shortest asymptotic input scale for strict greedy loss, with persistent deficit after a forced first merge.
Authors
- Kundan Baliga
- Aditya Byju
- Aiden Xie
Institutions
- Georgia Institute of Technology (US)
- University of Illinois Urbana-Champaign (US)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-17
- DOI
- https://doi.org/10.5281/zenodo.24324836
- Primary Topic
- Algorithms and Data Compression
- Type
- preprint