EST: A Morphology-Guided English-to-Sanskrit Tokenizer for Compact NLP Representations
We present EST, a tokenizer that maps English text into linguistically valid Sanskrit tokens, achieving 55.3 % average token reduction (peak 82 %) while preserving 95.2 % context. A 33 425-entry Paninian lexicon and a dual-path architecture ensure 100 % coverage with zero token loss. The system is released under an open licence for immediate reuse.
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2025-12-04
- DOI
- https://doi.org/10.5281/zenodo.17807017
- Primary Topic
- Natural Language Processing Techniques
- Type
- article
- Field-Weighted Citation Impact
- 0.00