Evaluating Artificial Intelligence Tools for Forensic-Linguistic Analysis of Arabic Judicial Texts: A Layer-Based Comparative Case Study of Saudi Judgments

Abstract This study evaluates GPT-4o (Omni; version 2024-05-13), supported by Arabic preprocessing with CAMeL Tools, against expert forensic-linguistic annotation of Saudi judicial judgments. Using a comparative evaluation design, the study analyzed 10 Saudi criminal judgments from Majmūʿat al-Aḥkām al-Qaḍāʾiyya (Ministry of Justice, 1435 AH/2013–2014 CE), from which 24 representative segments involving psychiatric evidence, legal capacity, and criminal responsibility were extracted. Each segment was coded across the five P-DIM layers, producing 120 segmentby-layer classification decisions. Two expert coders established the adjudicated benchmark after pre-adjudication reliability testing (88% agreement; pooled Cohen's kappa = .82). The AI arm used deterministic prompting (temperature = 0.0), a system-level forensic-linguistic persona, the full P- DIM codebook, and three few-shot gold-standard examples per layer. Results showed strong AI performance for formulaic and structurally explicit layers, especially legal formulae and institutional weighting, but weak performance for pragmatic-doctrinal interpretation. Layer 4 produced the main pragmatic bottleneck (F1 = .41; kappa = .28). The corrected fold-level ANOVA used 20 F1 observations (four validation folds for each of the five P-DIM layers) and showed a significant layer effect, F(4, 15) = 63.04, p = 3.367e-09, η² =.944. Tukey HSD confirmed that L4 was significantly lower than all other layers, while also showing additional significant non-L4 contrasts. Overall AI-expert agreement was fair (kappa = .275, N = 120), reflecting aggregate instability caused mainly by Layer 4. The study therefore supports a human-in-the-loop verification protocol rather than autonomous AI interpretation in Arabic forensic-linguistic analysis. Keywords: Arabic forensic linguistics; Saudi judicial judgments; AI-assisted annotation; expert coding; legal NLP evaluation; pragmatic-doctrinal inference; diglossia

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-10959403/v1
Primary Topic
Artificial Intelligence in Law
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Evaluating Artificial Intelligence Tools for Forensic-Linguistic Analysis of Arabic Judicial Texts: A Layer-Based Comparative Case Study of Saudi Judgments

Khaled Nasser Alfraidi, Bunder Sebail Alshammari, Mohammad Omar Aljudaiey, Walid Abdelhalim
1 links
Research Square
Artificial Intelligence in Law
preprint

Evaluating Artificial Intelligence Tools for Forensic-Linguistic Analysis of Arabic Judicial Texts: A Layer-Based Comparative Case Study of Saudi Judgments

Khaled Nasser Alfraidi, Bunder Sebail Alshammari, Mohammad Omar Aljudaiey, Walid Abdelhalim
preprint en

Abstract

Abstract This study evaluates GPT-4o (Omni; version 2024-05-13), supported by Arabic preprocessing with CAMeL Tools, against expert forensic-linguistic annotation of Saudi judicial judgments. Using a comparative evaluation design, the study analyzed 10 Saudi criminal judgments from Majmūʿat al-Aḥkām al-Qaḍāʾiyya (Ministry of Justice, 1435 AH/2013–2014 CE), from which 24 representative segments involving psychiatric evidence, legal capacity, and criminal responsibility were extracted. Each segment was coded across the five P-DIM layers, producing 120 segmentby-layer classification decisions. Two expert coders established the adjudicated benchmark after pre-adjudication reliability testing (88% agreement; pooled Cohen's kappa = .82). The AI arm used deterministic prompting (temperature = 0.0), a system-level forensic-linguistic persona, the full P- DIM codebook, and three few-shot gold-standard examples per layer. Results showed strong AI performance for formulaic and structurally explicit layers, especially legal formulae and institutional weighting, but weak performance for pragmatic-doctrinal interpretation. Layer 4 produced the main pragmatic bottleneck (F1 = .41; kappa = .28). The corrected fold-level ANOVA used 20 F1 observations (four validation folds for each of the five P-DIM layers) and showed a significant layer effect, F(4, 15) = 63.04, p = 3.367e-09, η² =.944. Tukey HSD confirmed that L4 was significantly lower than all other layers, while also showing additional significant non-L4 contrasts. Overall AI-expert agreement was fair (kappa = .275, N = 120), reflecting aggregate instability caused mainly by Layer 4. The study therefore supports a human-in-the-loop verification protocol rather than autonomous AI interpretation in Arabic forensic-linguistic analysis. Keywords: Arabic forensic linguistics; Saudi judicial judgments; AI-assisted annotation; expert coding; legal NLP evaluation; pragmatic-doctrinal inference; diglossia

Research Square
Beni-Suef University (EG), University of Ha'il (SA)
University of Hail
Peace, justice, and strong institutions
Artificial Intelligence in Law
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon