ArabAgentHallu: Towards Benchmarking Hallucination Judgment and Attribution in Arabic LLM Based Agents

Large language model (LLM) based agents execute multi step workflows in which hallucinations at intermediate steps propagate and corrupt final outcomes.Recent work formalized automated hallucination attribution identifying the trajectory step responsible and explaining its cause but existing benchmarks are exclusively English, where frontier models reach only 41.1% step localization accuracy.To the best of our knowledge, no Arabic benchmark evaluates hallucination attribution at the trajectory step level in LLM based agents.We present ArabAgentHallu, the first study of agent hallucination judgment and attribution in Arabic.Each trajectory is an executable program over a shared state, rendered into Arabic and English from one execution, with injected hallucinations and the responsible step verified counterfactually rather than asserted.The suite comprises 900 items (450 hallucinated / 450 clean) of 7-15 steps across five hallucination categories, balanced over a category × position × length grid.Evaluation is reported for two prompting conditions compared to three reference floors, with the bootstrapped intervals and paired cross lingual statistics appropriately sized to allow for parity claims to be falsifiable.With step localization accuracy 0.396 (Arabic) / 0.438 (English) and judgment accuracy 0.607 (Arabic) / 0.627 (English), the accuracy of the step localization is well above the ceiling of 0.164, while the judgment accuracy is relatively low.The paired Arabic-English difference is -0.042 [-0.132, +0.056]: no difference is detectable above the bound 0.135.With direct prompting the model collapses towards all negative strategy and a small significant gap towards English emerges: without the step by step scaffolding, the gap emerges.The code and generator are open and the data is open.

Authors

Institutions

Publication Details

Journal
International journal of intelligent engineering and systems
Published
2026-11-17
DOI
https://doi.org/10.22266/ijies2026.1231.45
Primary Topic
Natural Language Processing Techniques
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

ArabAgentHallu: Towards Benchmarking Hallucination Judgment and Attribution in Arabic LLM Based Agents

Nadia Mahmood Hussien, Salah Taha Allawi, Basim K. Abbas
International journal of intelligent engineering and systems
Natural Language Processing Techniques
article

ArabAgentHallu: Towards Benchmarking Hallucination Judgment and Attribution in Arabic LLM Based Agents

Nadia Mahmood Hussien, Salah Taha Allawi, Basim K. Abbas
article en

Abstract

Large language model (LLM) based agents execute multi step workflows in which hallucinations at intermediate steps propagate and corrupt final outcomes.Recent work formalized automated hallucination attribution identifying the trajectory step responsible and explaining its cause but existing benchmarks are exclusively English, where frontier models reach only 41.1% step localization accuracy.To the best of our knowledge, no Arabic benchmark evaluates hallucination attribution at the trajectory step level in LLM based agents.We present ArabAgentHallu, the first study of agent hallucination judgment and attribution in Arabic.Each trajectory is an executable program over a shared state, rendered into Arabic and English from one execution, with injected hallucinations and the responsible step verified counterfactually rather than asserted.The suite comprises 900 items (450 hallucinated / 450 clean) of 7-15 steps across five hallucination categories, balanced over a category × position × length grid.Evaluation is reported for two prompting conditions compared to three reference floors, with the bootstrapped intervals and paired cross lingual statistics appropriately sized to allow for parity claims to be falsifiable.With step localization accuracy 0.396 (Arabic) / 0.438 (English) and judgment accuracy 0.607 (Arabic) / 0.627 (English), the accuracy of the step localization is well above the ceiling of 0.164, while the judgment accuracy is relatively low.The paired Arabic-English difference is -0.042 [-0.132, +0.056]: no difference is detectable above the bound 0.135.With direct prompting the model collapses towards all negative strategy and a small significant gap towards English emerges: without the step by step scaffolding, the gap emerges.The code and generator are open and the data is open.

International journal of intelligent engineering and systemsVol. 19(12)
Mustansiriyah University (IQ)
Mustansiriyah University
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon