Akshara: Script-Aware, Grapheme-Cluster-Correct Array Operations for Indic Text
pandas, Polars, and Apache Arrow represent string columns as sequences of Unicode codepoints. For the nine Brahmic abugida scripts (Devanagari, Bengali, Gurmukhi, Gujarati, Odia, Tamil, Telugu, Kannada, Malayalam), a single user-perceived unit — an akshara, or orthographic syllable — is frequently encoded as a cluster of several codepoints, causing len() and slicing to silently corrupt conjuncts. This report introduces Akshara, a pure Python/NumPy library providing script-aware, grapheme-cluster-correct array operations — linear-time akshara segmentation, vectorized batch operations, and corpus pipeline integration — validated against 9,108 real FLORES-200 sentences across all nine target scripts with zero failures.
Authors
- Prashant Ahlawat (ORCID: https://orcid.org/0009-0005-7008-8567)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-07-30
- DOI
- https://doi.org/10.5281/zenodo.21697082
- Primary Topic
- Natural Language Processing Techniques
- Type
- preprint