ArabAgentHallu: Towards Benchmarking Hallucination Judgment and Attribution in Arabic LLM Based Agents
Large language model (LLM) based agents execute multi step workflows in which hallucinations at intermediate steps propagate and corrupt final outcomes.Recent work formalized automated hallucination attribution identifying the trajectory step responsible and explaining its cause but existing benchmarks are exclusively English, where frontier models reach only 41.1% step localization accuracy.To the best of our knowledge, no Arabic benchmark evaluates hallucination attribution at the trajectory step level in LLM based agents.We present ArabAgentHallu, the first study of agent hallucination judgment and attribution in Arabic.Each trajectory is an executable program over a shared state, rendered into Arabic and English from one execution, with injected hallucinations and the responsible step verified counterfactually rather than asserted.The suite comprises 900 items (450 hallucinated / 450 clean) of 7-15 steps across five hallucination categories, balanced over a category × position × length grid.Evaluation is reported for two prompting conditions compared to three reference floors, with the bootstrapped intervals and paired cross lingual statistics appropriately sized to allow for parity claims to be falsifiable.With step localization accuracy 0.396 (Arabic) / 0.438 (English) and judgment accuracy 0.607 (Arabic) / 0.627 (English), the accuracy of the step localization is well above the ceiling of 0.164, while the judgment accuracy is relatively low.The paired Arabic-English difference is -0.042 [-0.132, +0.056]: no difference is detectable above the bound 0.135.With direct prompting the model collapses towards all negative strategy and a small significant gap towards English emerges: without the step by step scaffolding, the gap emerges.The code and generator are open and the data is open.
Authors
- Nadia Mahmood Hussien (ORCID: https://orcid.org/0000-0002-2061-2149)
- Salah Taha Allawi (ORCID: https://orcid.org/0000-0002-5241-2504)
- Basim K. Abbas (ORCID: https://orcid.org/0000-0001-5572-2535)
Institutions
- Mustansiriyah University (IQ)
Publication Details
- Journal
- International journal of intelligent engineering and systems
- Published
- 2026-11-17
- DOI
- https://doi.org/10.22266/ijies2026.1231.45
- Primary Topic
- Natural Language Processing Techniques
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Mustansiriyah University