Arabic LLMs in Practice: Jais-3 vs. Qwen on Islamic Inheritance Law (Fara'id), Scriptural Traps and Tokenizer Economics

OVERVIEW An independent empirical audit comparing Arabic-centred open-weight models (Jais-3) with Qwen models and one commercial API reference on Islamic inheritance law (ʿilm al-farāʾiḍ, علم الفرائض), verbatim Quran and Hadith quotation, citation traps with numbers that do not exist, and tokenizer cost. All open-weight runs used one NVIDIA H200 GPU with vLLM 0.32.0 in bf16 on 25 October 2026. Scoring is rule-based (exact fractions in Python); no LLM acted as a judge. This is a sample evaluation with small $n$, not a benchmark, and it is not peer reviewed.https://github.com/sbeierle/arabic-llm-faraid MODELS inception42/Jais-3-32B (dense, 32B class) inception42/Jais-3-MoVA-36B-A4B (mixture of experts, 36B total, about 4B active) Qwen/Qwen2.5-32B-Instruct (dense, greedy) Qwen/Qwen3-30B-A3B (mixture of experts, 30.5B total, about 3.3B active) GPT-4o (OpenAI API, single run, reference only; the snapshot behind the identifier was not recorded) KEY FINDINGS Inheritance law (Fara'id, 50 tasks, answer key v2) Jais-3-32B and Jais-3-MoVA each solve 40 of 50 tasks (80%). Qwen3-30B-A3B solves 9 (18%), GPT-4o 12 (24%), and Qwen2.5-32B none. Within the mixture-of-experts pair, 32 tasks are solved only by Jais and 1 only by Qwen3 (exact McNemar test, $p = 7.9 \times 10^{-9}$). The gap holds with English prompts (35 vs. 6 of 50) and with a second seed (40 vs. 6 of 50). Six of the ten errors of each Jais model sit in the two ʿUmarī cases (al-ʿUmariyyatān). Both Jais models give the mother one third of the whole estate, the position attributed to Ibn ʿAbbās, while the answer key follows ʿUmar (one third of the remainder; Ibn Qudāma, al-Mughnī, vol. 6, around p. 172). Without these six tasks, both Jais models reach 40 of 44 (90.9%). Citation traps (50 numbers that do not exist: 20 hadith, 20 verse, 10 sura) The automatic scorer gave misleading results. After I read all 100 answers, the models do not differ: clean answers 29 of 50 for Qwen3 and 26 of 50 for Jais-3-MoVA (9 vs. 6 tasks, $p = 0.61$). For the 30 Quran verse and sura numbers, both models fail in 21 cases. The kind of error differs: Qwen3 often states correctly that the number does not exist and then adds a wrong replacement verse (15 of 50); Jais-3-MoVA more often attaches real verses to the wrong number or invents content (20 of 50). The categories were fixed after reading, so these tests are post hoc. Quran verse completion (40 verses) Jais-3-MoVA completes 31 of 40 with vowel marks and 26 of 40 without ($p = 0.30$). Qwen3-30B-A3B completes 0 of 40 in both conditions. Tokenizer cost (tokens per word, Qwen3 vs. Jais-3) Vocalised Quran text: 7.14 vs. 3.85 (1.85×) Sahih al-Bukhari: 6.04 vs. 3.17 (1.91×) Unvocalised Quran text: 1.51× English text: 1.00× Calculated from tokens per word, a fixed token budget holds about 52 to 72% as many Arabic words with the Qwen3 tokenizer. Memory use and serving cost were not measured. LIMITATIONS AND DISCLOSURE Prepared with Claude (Anthropic) as an agent under the author's direction. Claude wrote and revised the test code and the answer-key program, scored the runs and classified the trap answers. Mistakes happened and are listed in the report under Corrections; others may remain. The Fara'id answer keys were not reviewed by a scholar. The six rule variants are not assigned to schools. Each model ran with its model-card sampling values, not under equal conditions. Samples are small ($n = 20\text{ to }75$). This report is not a fatwā and must not be used to divide a real estate. DATA AND LICENSES The report text and results are released under CC BY 4.0. Prompts, scoring code and checksums are planned as a separate data package. It will contain only collection names and hadith numbers as references, no raw hadith text. The Quran text comes from the Tanzil project (CC BY 3.0, verbatim only); the unvocalised variant is a modification and will not be redistributed until the Tanzil terms are clarified.