Greedy byte-pair encoding asymptotically maximizes token savings on fixed alphabets

An explicit binary construction at the shortest asymptotic input scale for strict greedy loss, with persistent deficit after a forced first merge.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-17
DOI
https://doi.org/10.5281/zenodo.24324836
Primary Topic
Algorithms and Data Compression
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Greedy byte-pair encoding asymptotically maximizes token savings on fixed alphabets

Kundan Baliga, Aditya Byju, Aiden Xie
6 links
Zenodo (CERN European Organization for Nuclear Research)
Algorithms and Data Compression
preprint

Greedy byte-pair encoding asymptotically maximizes token savings on fixed alphabets

Kundan Baliga, Aditya Byju, Aiden Xie
preprint en

Abstract

An explicit binary construction at the shortest asymptotic input scale for strict greedy loss, with persistent deficit after a forced first merge.

Zenodo (CERN European Organization for Nuclear Research)
Georgia Institute of Technology (US), University of Illinois Urbana-Champaign (US)
Algorithms and Data Compression
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon