Fabricated Scripture: A Multilingual Audit of Citation and Verbatim-Text Hallucination in Large Language Models on Qur'an and Hadith Queries

Large language models (LLMs) are increasingly used for information retrieval across diverse domains, including religious knowledge. However, these models are prone to "hallucination"—generating confident but factually incorrect outputs. When such errors involve sacred texts like the Qur'an, the consequences extend beyond factual inaccuracy to potential theological harm, as users may unknowingly propagate fabricated scripture. We conducted a pilot study evaluating three open-source LLMs (qwen2.5:0.5b, llama3:latest, aya:8b) on Qur'an citation queries, testing both authentic verse retrieval and responses to "trap questions" requesting non-existent verses. Our findings reveal severe safety failures: all three models fabricated non-existent Qur'anic verses with high confidence (100% fabrication rate on trap questions, 5/5 tests), while only one model (33%) correctly retrieved a well-known authentic verse. Explicit cautious prompting had zero measurable effect on reducing fabrication. Multilingual evaluation on Bangla input resulted in catastrophic failures (0% success rate), including gibberish output and infinite repetition loops, contradicting aya:8b's claimed 101-language support. Paradoxically, the smallest model (494M parameters) outperformed larger models (8B parameters), suggesting training data quality matters more than parameter count for domain-specific accuracy. Our zero-cost methodology using local LLMs via Ollama demonstrates that systematic hallucination measurement is accessible without API costs. These findings establish that current open-source LLMs are unsafe for unverified Islamic knowledge queries and motivate immediate deployment of verification systems (retrieval-augmented generation, post-generation validation) in production AI systems handling religious content. We openly share our methodology, evaluation criteria, and pilot data to enable replication and extension to other religious traditions.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24270941
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Fabricated Scripture: A Multilingual Audit of Citation and Verbatim-Text Hallucination in Large Language Models on Qur'an and Hadith Queries

Najmun Nahar Hira
3 links
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Fabricated Scripture: A Multilingual Audit of Citation and Verbatim-Text Hallucination in Large Language Models on Qur'an and Hadith Queries

Najmun Nahar Hira
preprint en

Abstract

Large language models (LLMs) are increasingly used for information retrieval across diverse domains, including religious knowledge. However, these models are prone to "hallucination"—generating confident but factually incorrect outputs. When such errors involve sacred texts like the Qur'an, the consequences extend beyond factual inaccuracy to potential theological harm, as users may unknowingly propagate fabricated scripture. We conducted a pilot study evaluating three open-source LLMs (qwen2.5:0.5b, llama3:latest, aya:8b) on Qur'an citation queries, testing both authentic verse retrieval and responses to "trap questions" requesting non-existent verses. Our findings reveal severe safety failures: all three models fabricated non-existent Qur'anic verses with high confidence (100% fabrication rate on trap questions, 5/5 tests), while only one model (33%) correctly retrieved a well-known authentic verse. Explicit cautious prompting had zero measurable effect on reducing fabrication. Multilingual evaluation on Bangla input resulted in catastrophic failures (0% success rate), including gibberish output and infinite repetition loops, contradicting aya:8b's claimed 101-language support. Paradoxically, the smallest model (494M parameters) outperformed larger models (8B parameters), suggesting training data quality matters more than parameter count for domain-specific accuracy. Our zero-cost methodology using local LLMs via Ollama demonstrates that systematic hallucination measurement is accessible without API costs. These findings establish that current open-source LLMs are unsafe for unverified Islamic knowledge queries and motivate immediate deployment of verification systems (retrieval-augmented generation, post-generation validation) in production AI systems handling religious content. We openly share our methodology, evaluation criteria, and pilot data to enable replication and extension to other religious traditions.

Zenodo (CERN European Organization for Nuclear Research)
Shahjalal University of Science and Technology (BD)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon