Language and Script Gaps in Local Language Models for Indian Users

Many Indian users write to AI systems in an Indian language, often in Latin letters. I tested whether five openly released language models, run fully offline on one laptop, answer the same question less accurately in an Indian language or in romanized script (an Indian language written in Latin letters), and whether two simple fixes help. I used 300 parallel Belebele reading-comprehension questions in English and 13 Indian languages (four also romanized), collecting 76,400 answers in total, including Global-MMLU knowledge questions and a compression check. I tested five hypotheses (H1–H5) with exact McNemar tests and Holm correction for multiple testing. Accuracy was significantly lower than in English in all 65 model–language pairs (H1; gaps 7.0 to 35.7 percentage points (pp), mean 18.8 pp). Romanized input was significantly less accurate than native script in 15 of 20 pairs (H2; up to 31.3 pp), with Hindi the main exception. A language's token ratio relative to English was not significantly associated with its accuracy gap for any model (H3; 0 of 5). Converting romanized text to native script with IndicXlit raised accuracy significantly in 10 of 20 pairs (H4; up to 32.7 pp). Translating to English with NLLB-200 (1.3B) before answering helped significantly in only 3 of 65 pairs and lowered accuracy significantly in 9 (H5). Exploratory analyses found similar gaps on Global-MMLU, small drops for machine-made Indian English, and smaller but substantial gaps for an India-built model (Sarvam-30B). Results apply to compressed models, one task format and these specific fixes. Code, question lists, machine-made question versions and all model answers: https://github.com/omdorkar/local-language-models-india