Rule-Orbit Vocabulary (ROV)

Rule-Orbit Vocabulary (ROV): Lossless Vocabulary Factorization for Subword Tokenizers via Automatically Mined Rewrite Rules This record contains a preprint introducing Rule-Orbit Vocabulary (ROV), a method for reducing the embedding and softmax size of language models without changing tokenization. OverviewByte-pair-encoding (BPE) vocabularies devote many entries to surface variants of the same word that differ only by a simple rewrite, such as a leading space, capitalization or a common affix (e.g., walk, walks, walked, walking). Each variant needs its own embedding row and output logit, and rare variants are under-trained. Conventional vocabulary pruning removes such entries but lengthens token sequences. ROV takes a different approach. It keeps the original segmentation intact and re-identifies each token as a (base, rule) pair. Method Rewrite rules (prefix and suffix add/strip) are mined automatically from an existing tokenizer vocabulary. A distributional similarity gate admits a rule instance only when the paired tokens appear in similar contexts. Tokens are grouped into "orbits" around a canonical base form and encoded losslessly as (base_id, rule_id). The model uses an additive input embedding and a factorized output head, p(b | h) · p(r | b, h). Sequence length and bytes-per-token are unchanged by construction. Key findings In parameter-matched experiments on GPT-2-tokenized text across three small transformer sizes (128×2, 256×4, 512×8), case-based folding reduced loss in all 14 completed runs, by about 0.5–1.2% of total loss. The largest and most consistent gains were on rare variant tokens. These gains are partly confounded with a reallocation of parameters to wider feed-forward layers. Retrained 44k-entry tokenizers outperformed ROV by 0.025–0.044 bits per byte in a separate comparison. The paper reports these results transparently, positions ROV relative to concurrent work on compositional and case-aware tokenization, and outlines the controlled experiments needed to isolate the effect of folding. Contents ROV_Research_Paper.docx: full preprint ROV_Abstract.docx: one-page abstract StatusWorking paper and preprint. The results are preliminary, and further experiments are in progress.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24206592
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Rule-Orbit Vocabulary (ROV)

Ujjwalkumar Bharatkumar Soni
1 links
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Rule-Orbit Vocabulary (ROV)

Ujjwalkumar Bharatkumar Soni
preprint en

Abstract

Rule-Orbit Vocabulary (ROV): Lossless Vocabulary Factorization for Subword Tokenizers via Automatically Mined Rewrite Rules This record contains a preprint introducing Rule-Orbit Vocabulary (ROV), a method for reducing the embedding and softmax size of language models without changing tokenization. OverviewByte-pair-encoding (BPE) vocabularies devote many entries to surface variants of the same word that differ only by a simple rewrite, such as a leading space, capitalization or a common affix (e.g., walk, walks, walked, walking). Each variant needs its own embedding row and output logit, and rare variants are under-trained. Conventional vocabulary pruning removes such entries but lengthens token sequences. ROV takes a different approach. It keeps the original segmentation intact and re-identifies each token as a (base, rule) pair. Method Rewrite rules (prefix and suffix add/strip) are mined automatically from an existing tokenizer vocabulary. A distributional similarity gate admits a rule instance only when the paired tokens appear in similar contexts. Tokens are grouped into "orbits" around a canonical base form and encoded losslessly as (base_id, rule_id). The model uses an additive input embedding and a factorized output head, p(b | h) · p(r | b, h). Sequence length and bytes-per-token are unchanged by construction. Key findings In parameter-matched experiments on GPT-2-tokenized text across three small transformer sizes (128×2, 256×4, 512×8), case-based folding reduced loss in all 14 completed runs, by about 0.5–1.2% of total loss. The largest and most consistent gains were on rare variant tokens. These gains are partly confounded with a reallocation of parameters to wider feed-forward layers. Retrained 44k-entry tokenizers outperformed ROV by 0.025–0.044 bits per byte in a separate comparison. The paper reports these results transparently, positions ROV relative to concurrent work on compositional and case-aware tokenization, and outlines the controlled experiments needed to isolate the effect of folding. Contents ROV_Research_Paper.docx: full preprint ROV_Abstract.docx: one-page abstract StatusWorking paper and preprint. The results are preliminary, and further experiments are in progress.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon