SpearIndic: A Lossless Tokenizer for Native-Script, Romanized and Code-Mixed Text in Indian Languages

Indian users write the same language in three ways: in its native script, romanized in Latin letters, and code-mixed with English. Widely used tokenizers are built for native script or for English, so the other styles are split into many tokens. We train SentencePiece Unigram and BPE tokenizers with byte fallback on one corpus with native-script text for 22 scheduled Indian languages, romanized and code-mixed text for 11 of them, and English. We evaluate them on a test set of 97,750 texts that was frozen before any training data was sampled. With a 20% smaller vocabulary than IndicBERTv2, our 200k Unigram model compresses native-script text better overall (+4.7% bytes per token on IN22-Gen, +1.0% on FLORES-200) and code-mixed text better (+2.0%), while being lossless on every test text; IndicBERTv2, as released, reproduces fewer than 1% of them. It is the most compact lossless tokenizer in our comparison on every Indic test set, using less than half the tokens of GPT-4o's o200k on native script. The native-script lead is driven by low-resource languages, and on independent human romanizations (Dakshina) it is on par with IndicBERTv2 rather than ahead. Controlled ablations show that Unigram needs 1.7-2.7% fewer tokens than BPE on Indic text, that each doubling of the vocabulary from 16k to 64k raises bytes per token by 14-16% (12-14% fewer tokens), and that covering 22 instead of 5 languages at a fixed 64k vocabulary costs the Dravidian languages about 20% more tokens. We also report a Unicode pitfall: training on NFC-normalized text while encoding raw text turns common precomposed letters into byte tokens. Forcing full script coverage removes this problem without giving up losslessness. Tokenizer (Hugging Face): https://huggingface.co/SpearsoftLabs/spearindic-tokenizer