OdiaSLM: From-Scratch Pretraining, Morphology-Aware Tokenization, and Calibrated Evaluation of 60M and 100M Small Language Models for Odia

While frontier multilingual Large Language Models (LLMs) impose a severe tokenization tax on morphologically rich, low-resource Indic scripts such as Odia (ory_Orya)—fragmenting words into 5-6x more subword tokens than English and inflating inference latency—compact, script-native models remain underexplored. We present OdiaSLM-60M (62.11M parameters) and OdiaSLM-100M (100.69M parameters), two sovereign decoder-only Small Language Models pre-trained from scratch for Odia on up to 1.57 billion tokens drawn from a strictly deduplicated and 13-gram benchmark-decontaminated corpus of 9.62 million documents (1.247 billion words). First, we design a 16,384-vocabulary canonical NFC-normalized SentencePiece BPE tokenizer that preserves Odia juktakshara conjuncts, achieving a fertility of 1.4809 tokens/word (11.57 UTF-8 bytes/token), 0 <unk> tokens, and a 3.6x compression advantage over general multilingual tokenizers. Second, utilizing Grouped-Query Attention (GQA), SwiGLU, Half-Split RoPE, FP32-variance RMSNorm, auxiliary Z-loss stabilization, a Warmup-Stable-Decay (WSD) schedule, and two-stage context extension (512 to 1024), we demonstrate clean neural scaling from 62.11M (validation perplexity 19.92, Bits-Per-Byte 0.3965) to 100.69M parameters (validation perplexity 17.14, Bits-Per-Byte 0.3767, Top-1 token accuracy 44.61%). Third, we conduct a comprehensive zero-shot evaluation across 27 tasks in the Odia Evaluation Benchmark, uncovering both cross-lingual untranslated choice artifacts in 9 tasks and severe surface-form competition under raw log-likelihood. Applying Pointwise Mutual Information (PMI) calibration unlocks a macro-average accuracy of 60.83% across the 18 pure-Odia benchmarks (+34.44% lift over random chance). Both models are released in native LlamaForCausalLM format.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-12
DOI
https://doi.org/10.5281/zenodo.24167432
Primary Topic
Natural Language Processing Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

OdiaSLM: From-Scratch Pretraining, Morphology-Aware Tokenization, and Calibrated Evaluation of 60M and 100M Small Language Models for Odia

Arun Misra
Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
preprint

OdiaSLM: From-Scratch Pretraining, Morphology-Aware Tokenization, and Calibrated Evaluation of 60M and 100M Small Language Models for Odia

Arun Misra
preprint en

Abstract

While frontier multilingual Large Language Models (LLMs) impose a severe tokenization tax on morphologically rich, low-resource Indic scripts such as Odia (ory_Orya)—fragmenting words into 5-6x more subword tokens than English and inflating inference latency—compact, script-native models remain underexplored. We present OdiaSLM-60M (62.11M parameters) and OdiaSLM-100M (100.69M parameters), two sovereign decoder-only Small Language Models pre-trained from scratch for Odia on up to 1.57 billion tokens drawn from a strictly deduplicated and 13-gram benchmark-decontaminated corpus of 9.62 million documents (1.247 billion words). First, we design a 16,384-vocabulary canonical NFC-normalized SentencePiece BPE tokenizer that preserves Odia juktakshara conjuncts, achieving a fertility of 1.4809 tokens/word (11.57 UTF-8 bytes/token), 0 <unk> tokens, and a 3.6x compression advantage over general multilingual tokenizers. Second, utilizing Grouped-Query Attention (GQA), SwiGLU, Half-Split RoPE, FP32-variance RMSNorm, auxiliary Z-loss stabilization, a Warmup-Stable-Decay (WSD) schedule, and two-stage context extension (512 to 1024), we demonstrate clean neural scaling from 62.11M (validation perplexity 19.92, Bits-Per-Byte 0.3965) to 100.69M parameters (validation perplexity 17.14, Bits-Per-Byte 0.3767, Top-1 token accuracy 44.61%). Third, we conduct a comprehensive zero-shot evaluation across 27 tasks in the Odia Evaluation Benchmark, uncovering both cross-lingual untranslated choice artifacts in 9 tasks and severe surface-form competition under raw log-likelihood. Applying Pointwise Mutual Information (PMI) calibration unlocks a macro-average accuracy of 60.83% across the 18 pure-Odia benchmarks (+34.44% lift over random chance). Both models are released in native LlamaForCausalLM format.

Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon