Is Romanization Still a Discount? Measuring the Token Cost of Human-Romanized Hindi Across Tokenizer Generations

Many Hindi speakers type in Latin script, and prior work reports that romanization cuts token counts for Indic languages by 2 to 4 times. Those estimates come from machine transliteration and the Llama 2 tokenizer. We measure the token cost of the same 8,847 Hindi sentences in four forms: native Devanagari, human romanization from the Dakshina dataset, and two machine romanizations (IndicXlit and ITRANS). We cover 12 open tokenizers whose vocabularies were released between 2019 and 2025. Machine romanization overstates the cost of romanized Hindi: human-romanized text needs 11.5% fewer tokens than IndicXlit output on average (8.9% to 13.8% across tokenizers) and 9.4% fewer than ITRANS. The romanization discount over Devanagari depends on the tokenizer. It is 2.4x to 3.7x for the four tokenizers that fragment Devanagari most (GPT-2, cl100k_base, Llama 2 and Qwen3), but for 6 of 12, including o200k_base, Gemma 3 and Llama 4, romanized Hindi needs more tokens than Devanagari (0.54x to 0.99x). Within the Llama family the discount falls from 2.52x to 1.38x to 0.94x. Tokenizer rankings barely change between human and machine romanization (Kendall's tau = 0.94). We release our code and per-sentence counts. Code and per-sentence token counts: https://github.com/Pallavrai/romanized_discount_research