Tokenization Disparity in Kazakh: Cross-Tokenizer Measurement and Low-Cost Vocabulary Adaptation for Large Language Models

Large language models are billed by the token, and tokens are not distributed evenly across languages. We measure how many tokens ten widely deployed tokenizers require to encode identical content in Kazakh and English, using 900 sentence-aligned pairs from OPUS-100 with bootstrap confidence intervals over 10,000 resamples. Kazakh requires 3.16x more tokens than English under GPT-4's tokenizer and 2.62x under Qwen2.5-7B. This corresponds to a 194% premium in processing cost and a context window holding 38% as much material. We show this is not a property of the Kazakh language. On the same sentences XLM-R needs 1.32x and Aya-101 needs 1.31x. The clearest evidence comes from one provider: GPT-4 requires 3.16x while GPT-4o, one generation later, requires 1.75x — a 44% reduction achieved purely by changing the tokenizer. Vocabulary size does not predict the premium: mBERT achieves a lower ratio than Qwen2.5 with a smaller vocabulary. We then test vocabulary extension as a remedy. Adding 5,875 Kazakh tokens to Qwen2.5-0.5B and training only the new embeddings under a single continuous 6,000-step schedule yields 1.718x compression with a 19.81% improvement in bits per character in domain, at 1.2% additional parameters and 74 minutes on a free-tier GPU. On out-of-domain Kazakh web text the compression persists at 1.591x and the quality cost falls to 2.53%, close to parity, having been 38.08% at 1,000 steps. The out-of-domain penalty is therefore substantially an undertraining artifact rather than domain overfitting. On KazMMLU (931 Kazakh questions, chance 23.9%) the extended model scores 27.8% against a baseline of 23.3%; a paired McNemar test over the discordant answers gives p = 0.0152. The checkpoint was selected on out-of-domain bits per character before the benchmark was run. An ablation at a fixed 2,000-step budget shows subword-composition initialisation reaching 14.12% where random initialisation reaches 7.38%, so roughly half the gain comes from the initialisation method rather than the added parameters.