Roman Urdu Breaks Dense Retrieval: Script, Spelling Variation and a Transliteration Fix

Much everyday Urdu is typed in the Latin script (Roman Urdu), with no standard spelling, while the text people want to search is written in Urdu script. We measure what this does to retrieval. On a new set of 200 test questions over 3,674 Wikipedia passages, each question asked in Urdu script, in English and in three Roman Urdu spellings, every dense embedder we test loses between 29 and 59 points of Recall@5 when the question moves from Urdu script to Roman Urdu, and BM25 finds almost nothing. A second test with human-written Roman Urdu questions from Belebele shows a similar gap (40 to 51 points), although there the Roman Urdu is also worded differently. Spelling variation is a smaller effect than script, but for up to 14% of questions one spelling finds the answer and another misses it. Rule-based spelling normalization makes things worse. Transliterating the query into Urdu script before searching closes most of the gap: with an LLM transliterator the best embedders come within a few points of their Urdu-script score, and a small open transliteration model recovers 90% of the LLM's gain without an API call, while a general 7B open LLM does markedly worse. We release the question set and code: https://github.com/masooddahmedd/roman-urdu-retrieval

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-20
DOI
https://doi.org/10.5281/zenodo.24384588
Primary Topic
Information Retrieval and Search Behavior
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Roman Urdu Breaks Dense Retrieval: Script, Spelling Variation and a Transliteration Fix

Masood Ahmed
1 links
Zenodo (CERN European Organization for Nuclear Research)
Information Retrieval and Search Behavior
preprint

Roman Urdu Breaks Dense Retrieval: Script, Spelling Variation and a Transliteration Fix

Masood Ahmed
preprint en

Abstract

Much everyday Urdu is typed in the Latin script (Roman Urdu), with no standard spelling, while the text people want to search is written in Urdu script. We measure what this does to retrieval. On a new set of 200 test questions over 3,674 Wikipedia passages, each question asked in Urdu script, in English and in three Roman Urdu spellings, every dense embedder we test loses between 29 and 59 points of Recall@5 when the question moves from Urdu script to Roman Urdu, and BM25 finds almost nothing. A second test with human-written Roman Urdu questions from Belebele shows a similar gap (40 to 51 points), although there the Roman Urdu is also worded differently. Spelling variation is a smaller effect than script, but for up to 14% of questions one spelling finds the answer and another misses it. Rule-based spelling normalization makes things worse. Transliterating the query into Urdu script before searching closes most of the gap: with an LLM transliterator the best embedders come within a few points of their Urdu-script score, and a small open transliteration model recovers 90% of the LLM's gain without an API call, while a general 7B open LLM does markedly worse. We release the question set and code: https://github.com/masooddahmedd/roman-urdu-retrieval

Zenodo (CERN European Organization for Nuclear Research)
Information Retrieval and Search Behavior
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Roman Urdu Breaks Dense Retrieval: Script, Spelling Variation and a Transliteration Fix — Masood Ahmed · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS