Comprehension, Not Only Compliance: How Denominators and Detectors Shape Cross-Lingual Safety Gaps for Urdu, Roman Urdu and Code-Switched Prompts

Multilingual safety evaluations usually summarise a model by a single “safety gap” betweenEnglish and another language. We re-analyse a released evaluation, VeriAudit v0.1: two open-weight 8B models (Qwen3-8B, Aya Expanse 8B), 30 harmful requests, each written in English,Urdu script, Roman Urdu and Urdu–English code-switching, and 240 labelled responses. Wefind that the gap depends on two choices that are rarely reported. First, the denominator.Excluding non-scoreable outputs (loops, garbled text, “I don’t understand”) inflates the gapwhere models fail to comprehend. Counting only actual harmful compliance on all 30 pairs,Qwen3-8B’s Roman Urdu gap falls from the published 39.5 to 26.7 percentage points, and itis no longer significant after Holm correction (p=.086). Treating the same outputs as unsaferaises it to 56.7 points. Second, the detector. The evaluation’s English refusal heuristic found 0%of Urdu-script refusals and 16–19% of Roman Urdu and code-switched refusals, against 92% inEnglish. On the same responses it would have reported gaps of 42.5–70 points. We reproduce all191 published numbers of v0.1 exactly from raw outputs. An exploratory follow-up adds a newermodel (Qwen3.5-9B), three sampled responses per prompt and two independent automatedjudges. It reproduces the response-language drift and the direction of the largest gaps, andshows that the judges agree on harmfulness far more often than on the type of refusal. Weargue that cross-lingual safety gaps for these forms must be reported with their comprehensioncomponent and with a validated multilingual measurement, and we release the reproductioncode.

Authors

Publication Details

Journal
Figshare
Published
2026-11-19
DOI
https://doi.org/10.6084/m9.figshare.35063674
Primary Topic
Hate Speech and Cyberbullying Detection
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Comprehension, Not Only Compliance: How Denominators and Detectors Shape Cross-Lingual Safety Gaps for Urdu, Roman Urdu and Code-Switched Prompts

Abeera Najam
Figshare
Hate Speech and Cyberbullying Detection
preprint

Comprehension, Not Only Compliance: How Denominators and Detectors Shape Cross-Lingual Safety Gaps for Urdu, Roman Urdu and Code-Switched Prompts

Abeera Najam
preprint en

Abstract

Multilingual safety evaluations usually summarise a model by a single “safety gap” betweenEnglish and another language. We re-analyse a released evaluation, VeriAudit v0.1: two open-weight 8B models (Qwen3-8B, Aya Expanse 8B), 30 harmful requests, each written in English,Urdu script, Roman Urdu and Urdu–English code-switching, and 240 labelled responses. Wefind that the gap depends on two choices that are rarely reported. First, the denominator.Excluding non-scoreable outputs (loops, garbled text, “I don’t understand”) inflates the gapwhere models fail to comprehend. Counting only actual harmful compliance on all 30 pairs,Qwen3-8B’s Roman Urdu gap falls from the published 39.5 to 26.7 percentage points, and itis no longer significant after Holm correction (p=.086). Treating the same outputs as unsaferaises it to 56.7 points. Second, the detector. The evaluation’s English refusal heuristic found 0%of Urdu-script refusals and 16–19% of Roman Urdu and code-switched refusals, against 92% inEnglish. On the same responses it would have reported gaps of 42.5–70 points. We reproduce all191 published numbers of v0.1 exactly from raw outputs. An exploratory follow-up adds a newermodel (Qwen3.5-9B), three sampled responses per prompt and two independent automatedjudges. It reproduces the response-language drift and the direction of the largest gaps, andshows that the judges agree on harmfulness far more often than on the type of refusal. Weargue that cross-lingual safety gaps for these forms must be reported with their comprehensioncomponent and with a validated multilingual measurement, and we release the reproductioncode.

Figshare
Hate Speech and Cyberbullying Detection
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon