Interpreting isiZulu Idioms with Large Language Models: A Comparative Study of Zero-Shot Prompting, Retrieval-Augmented Generation, and Supervised Fine-Tuning

Idiom interpretation requires cultural knowledge, context, and pragmatic understanding that cannot be inferred from literal word meanings alone. This challenge is particularly important for isiZulu, a richly inflected Nguni Bantu language with limited representation in the web-scale corpora used to train modern large language models (LLMs). We compare three approaches to isiZulu idiom interpretation (zero-shot prompting, retrieval-augmented generation (RAG) with glossary evidence, and supervised fine-tuning on a purpose-built, context-rich corpus) across five open-weight LLMs, evaluating them on 323 held-out idioms. Performance is evaluated using automated metrics and human assessment by expert isiZulu speakers. Supervised fine-tuning provides the strongest overall improvement, increasing the mean human evaluation score from 2.61 to 3.68 out of 5. RAG also improves interpretation quality, although its gains are smaller and are not fully captured by conventional surface-level metrics. Across the fifteen model--condition systems, COMET aligns best with human assessment, whereas embedding cosine similarity is the only metric that does not align positively with it, suggesting that this metric should be validated against human judgement before being used to evaluate idiomatic interpretation in low-resource languages. We contribute the isiZulu Idiom Dataset, a morphology-aware retrieval pipeline, and a reusable protocol for evaluating LLM-based idiom interpretation in low-resource African languages.

Authors

Institutions

Publication Details

Journal
Journal of the Digital Humanities Association of Southern Africa
Published
2026-11-20
DOI
https://doi.org/10.55492/dhasa.v7i02.7708
Primary Topic
Natural Language Processing Techniques
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Interpreting isiZulu Idioms with Large Language Models: A Comparative Study of Zero-Shot Prompting, Retrieval-Augmented Generation, and Supervised Fine-Tuning

Sthembiso Mkhwanazi, MARIVATE Vukosi, Joseph Sefara, Nontokozo Buhle Ntandoyenkosi Manukuza et al.
3 links
Journal of the Digital Humanities Association of Southern Africa
Natural Language Processing Techniques
article

Interpreting isiZulu Idioms with Large Language Models: A Comparative Study of Zero-Shot Prompting, Retrieval-Augmented Generation, and Supervised Fine-Tuning

Sthembiso Mkhwanazi, MARIVATE Vukosi, Joseph Sefara, Nontokozo Buhle Ntandoyenkosi Manukuza, Kwanele R
article en

Abstract

Idiom interpretation requires cultural knowledge, context, and pragmatic understanding that cannot be inferred from literal word meanings alone. This challenge is particularly important for isiZulu, a richly inflected Nguni Bantu language with limited representation in the web-scale corpora used to train modern large language models (LLMs). We compare three approaches to isiZulu idiom interpretation (zero-shot prompting, retrieval-augmented generation (RAG) with glossary evidence, and supervised fine-tuning on a purpose-built, context-rich corpus) across five open-weight LLMs, evaluating them on 323 held-out idioms. Performance is evaluated using automated metrics and human assessment by expert isiZulu speakers. Supervised fine-tuning provides the strongest overall improvement, increasing the mean human evaluation score from 2.61 to 3.68 out of 5. RAG also improves interpretation quality, although its gains are smaller and are not fully captured by conventional surface-level metrics. Across the fifteen model--condition systems, COMET aligns best with human assessment, whereas embedding cosine similarity is the only metric that does not align positively with it, suggesting that this metric should be validated against human judgement before being used to evaluate idiomatic interpretation in low-resource languages. We contribute the isiZulu Idiom Dataset, a morphology-aware retrieval pipeline, and a reusable protocol for evaluating LLM-based idiom interpretation in low-resource African languages.

Journal of the Digital Humanities Association of Southern AfricaVol. 7(2)
Impact (CA), University of Pretoria (ZA)
Nvidia, Government of the United Kingdom, International Development Research Centre
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Interpreting isiZulu Idioms with Large Language Models: A Comparative Study of Zero-Shot Prompting, Retrieval-Augmented Generation, and Supervised Fine-Tuning — Sthembiso Mkhwanazi, MARIVATE Vukosi, et al. · Journal of the Digital Humanities Association of Southern Africa (2026) | TGRS Research Map | TGRS