Banya AI 5: Cache-Dictionary Tokenization and Dynamic Embedding , Near-Free Vocabulary Expansion via a Low-Rank Head

Tokenization — splitting text into the word units a model handles — is arranged in two layers. The lower layer consists of syllable atoms (Korean phonetic units), and the upper layer is a bundle dictionary that holds frequent words as whole tokens. The vocabulary of the measured model stacks 8000 bundles on top of 2724 syllable atoms, for a total of 10724. Ordinarily, increasing the vocabulary fourfold makes the output head (the final layer that selects the next word) proportionally more expensive in parameters, but this paper makes vocabulary expansion nearly free by giving the output head a low-rank form (approximating the large matrix as a product of two smaller matrices). At vocabulary 10724, channel width 1024, and rank 128, the low-rank head has about 1.5 million parameters, about 7.3 times fewer than the dense head's about 10.98 million. Furthermore, bundle-layer tokens preserve the orthogonal structure of the data plane and the operation plane intact (the operator emergence of Part 3 also appears in bundles, with a direction consistency of 8.1 times). That is, the cache dictionary gains compression without breaking the data-operation structure established by the earlier papers. We also measured that the savings travel together with quality. The normal path, bundles on, predicts holdout text at 5.08 bits per character while encoding the same text with 21.5% fewer tokens; with bundles off, prediction collapses at word-internal positions (11.49 bits per token vs. 6.69 at boundaries), revealing that this system's word knowledge resides in the bundle path. This record contains the English edition (_en.pdf) and the Korean original (_kr.pdf). Part 5 of the six-part Banya AI research series. Part 1: https://doi.org/10.5281/zenodo.21385447. Part 2: https://doi.org/10.5281/zenodo.21385516. Part 3: https://doi.org/10.5281/zenodo.21385676. Part 4: https://doi.org/10.5281/zenodo.21385699. All measured figures are reproducible with the public code package: https://ubmscoin.github.io/banya/banya_ai_paper_code/ (code Apache-2.0). Frozen model checkpoints: https://doi.org/10.5281/zenodo.21383724