Reading the Meaning, Missing the Language: AI and Human Agreement on Nigerian Public Discourse.

What we did. We report a reliability study and a multi-model evaluation of meaning in Nigerian public discourse. It uses the Meaning Intelligence Framework (MIF™), a nine-dimension schema that separates surface sentiment from communicative intent. Five Certified Meaning Intelligence Analysts independently triple-annotated 500 naturalistic social-media, news and speech items in Standard English, Nigerian English, Nigerian Pidgin and code-mixed speech, for 1,500 annotations. How far trained readers agree. No dimension reached Krippendorff's α = 0.667. Agreement was highest on register (0.60) and lowest on risk tier (0.24). Analysts certified at 85–100% on constructed items agreed far less on real ones. What we tested. We built a two-tier gold standard and evaluated six models from OpenAI, Google and Anthropic, scoring each against the same analyst pairs as a leave-one-out human baseline. A second expert, blind to the first adjudicator's labels, re-adjudicated 50 contested items. The two experts agreed on 78–82% of register, sentiment, irony and risk labels, but on only 56% of recommended actions. Where models stand. Models matched analysts on intent family (0.73–0.77 against 0.76) and exceeded them on subtext presence. They fell short on recommended action, though part of that gap reflects contested expert judgement: against the second adjudicator, three models' action agreement rose by 0.18–0.24. All models fell short on register. Analysts recognised Nigerian Pidgin at 0.92; the best models reached 0.59–0.60, and Gemini models 0.11–0.15. What changed the result. Expert knowledge from the MIF changed the register result. A sharpened MIF register definition, which describes Pidgin by its grammatical markers, raised model Pidgin recognition from 0.20–0.47 to 0.90–0.96, level with the analysts. It came at a cost to code-mixed accuracy and awaits confirmation on held-out items. In a controlled probe, OpenAI models refused three benign Pidgin and code-mixed posts in 6 of 6 calls, and answered their Standard English renderings in 6 of 6. Conclusion. Current models increasingly read Nigerian meaning but misrecognise Nigerian language varieties. A small amount of expert knowledge closes much of that gap, and responsible deployment requires variety-level auditing.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24217238
Primary Topic
Sentiment Analysis and Opinion Mining
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Reading the Meaning, Missing the Language: AI and Human Agreement on Nigerian Public Discourse.

Celestine Ngozichukwuka Achi
Zenodo (CERN European Organization for Nuclear Research)
Sentiment Analysis and Opinion Mining
preprint

Reading the Meaning, Missing the Language: AI and Human Agreement on Nigerian Public Discourse.

Celestine Ngozichukwuka Achi
preprint en

Abstract

What we did. We report a reliability study and a multi-model evaluation of meaning in Nigerian public discourse. It uses the Meaning Intelligence Framework (MIF™), a nine-dimension schema that separates surface sentiment from communicative intent. Five Certified Meaning Intelligence Analysts independently triple-annotated 500 naturalistic social-media, news and speech items in Standard English, Nigerian English, Nigerian Pidgin and code-mixed speech, for 1,500 annotations. How far trained readers agree. No dimension reached Krippendorff's α = 0.667. Agreement was highest on register (0.60) and lowest on risk tier (0.24). Analysts certified at 85–100% on constructed items agreed far less on real ones. What we tested. We built a two-tier gold standard and evaluated six models from OpenAI, Google and Anthropic, scoring each against the same analyst pairs as a leave-one-out human baseline. A second expert, blind to the first adjudicator's labels, re-adjudicated 50 contested items. The two experts agreed on 78–82% of register, sentiment, irony and risk labels, but on only 56% of recommended actions. Where models stand. Models matched analysts on intent family (0.73–0.77 against 0.76) and exceeded them on subtext presence. They fell short on recommended action, though part of that gap reflects contested expert judgement: against the second adjudicator, three models' action agreement rose by 0.18–0.24. All models fell short on register. Analysts recognised Nigerian Pidgin at 0.92; the best models reached 0.59–0.60, and Gemini models 0.11–0.15. What changed the result. Expert knowledge from the MIF changed the register result. A sharpened MIF register definition, which describes Pidgin by its grammatical markers, raised model Pidgin recognition from 0.20–0.47 to 0.90–0.96, level with the analysts. It came at a cost to code-mixed accuracy and awaits confirmation on held-out items. In a controlled probe, OpenAI models refused three benign Pidgin and code-mixed posts in 6 of 6 calls, and answered their Standard English renderings in 6 of 6. Conclusion. Current models increasingly read Nigerian meaning but misrecognise Nigerian language varieties. A small amount of expert knowledge closes much of that gap, and responsible deployment requires variety-level auditing.

Zenodo (CERN European Organization for Nuclear Research)
Sentiment Analysis and Opinion Mining
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon