Tokenizer Benchmarking Across Indic and Code‑Mixed Scripts

Tokenizer Benchmarking Across Indic and Code-Mixed Scripts Tokenization represents a critical, irreversible step in Large Language Model (LLM) pipelines, constraining context budgets, inference costs, and model robustness, particularly for noisy or code-mixed inputs. This deposit contains the data, code, and supplementary materials for the paper "Tokenizer Benchmarking Across Indic and Code-Mixed Scripts." We systematically compare 11 production-grade tokenizers (including models from OpenAI, Meta, and Mistral, alongside Indic-specialized WordPiece/SentencePiece models) across monolingual English, three Indic scripts, and Hinglish (Devanagari+Latin code-mixing). All metrics including tokens-per-100-chars, bytes-per-token, [UNK] incidence, and fragmentation scores were computed using a reproducible browser-based harness and exported as raw CSVs. Deposit Contents main.pdf: Final compiled research paper release-v1.0.1.zip: Complete source archive including main.tex, refs.bib, and LaTeX assets tokenizer-output/: Raw measurement data exported as CSVs (tracked via Git LFS in GitHub repository) tokenizer-input/: Curated baseline input pools used for sampling baselines/: Template IDs enabling exact re-sampling of the 100 baselines per slice analysis/: Python pipeline used to generate all figures and tables from raw CSV data figures/ & tables/: Generated assets used in the manuscript sha256sum.csv: SHA-256 checksums for verifying large file integrity Documentation: README.md, DATA_README.md, CITATION.cff, and license files Reproducibility (Quick Start) Clone the repository or download the release ZIP Verify data integrity: powershell Get-FileHash -Algorithm SHA256 tokenizer-output\english-fast.csv Compare output with checksums/sha256sum.csv Run the analysis pipeline: powershell python -m venv .venv . .venv/Scripts/Activate.ps1 pip install -r analysis/requirements.txt python analysis/generate_figures.py Refer to Appendix D in the paper for detailed documentation on the CSV-first analysis pipeline. Citation Concept DOI: 10.5281/zenodo.17273988 (resolves to latest version) Suggested Citation: Likheet Shetty (2025). Tokenizer Benchmarking Across Indic and Code-Mixed Scripts. Zenodo. https://doi.org/10.5281/zenodo.17273988 Licensing and Reuse Paper/Text: CC-BY 4.0 (see LICENSE.txt) Code: MIT License (see LICENSE.txt) Data: See LICENSE-data.txt for dataset-specific terms Privacy Note: Baseline pools were PII-scrubbed prior to release. See Appendix B for curation details. Technical Notes Large files: CSV artifacts are approximately 50-60 MB each Harness details: Browser harness utilized transformers.js and tiktoken (WebAssembly). Specific versions and commit SHAs are logged in every CSV row for exact reproducibility. Related Resources GitHub Repository: https://github.com/Likheet/indic-tokenizer-paper TokenizerLab: https://github.com/Likheet/tokenizer-lab (browser app used for measurements) Contact Likheet Shetty ([email protected])