Evaluating Robustness of LLMs to Typographical Noise in Yorùbá QA

Generative AI models are primarily accessed through chat interfaces, where user queries often contain typographical errors.While these models perform well in English, their robustness to noisy inputs in low-resource languages like Yorb remains underexplored.This work investigates a Yorb question-answering (QA) task by introducing synthetic typographical noise into clean inputs.We design a probabilistic noise injection strategy that simulates realistic human typos.In our experiments, each character in a clean sentence is independently altered, with noise levels ranging from 10% to 40%.We evaluate performance across three strong multilingual models using two complementary metrics: (1) a multilingual BERTScore to assess semantic similarity between outputs on clean and noisy inputs, and (2) an LLM-asjudge approach, where the best Yorb-capable model rates fluency, comprehension, and accuracy on a 1-5 scale.Results show that while English QA performance degrades gradually, Yorb QA suffers a sharper decline.At 40% noise, GPT-4o experiences over a 50% drop in comprehension ability, with similar declines for Gemini 2.0 Flash and Claude 3.7 Sonnet.We conclude with recommendations for noiseaware training and dedicated noisy Yorb benchmarks to enhance LLM robustness in lowresource settings.