Polite to Whom? Benchmarking Korean Speech-Level Maintenance and Switching in Multi-Turn LLM Conversations

A Korean assistant chooses how to address its listener in every reply, and the right speech level can change mid-conversation. We introduce a benchmark that tests this choice across complete conversations that each model generates for itself. Its 250 fictional scripts cover keeping an assigned level, switching on request, switching when a new listener arrives, choosing a level from the relationship alone, and honorifying third parties. Five API models produced 17,500 replies, and three human reviewers verified a PASS/FAIL verdict for each. Mean reply success per dialogue is 97.8% to 99.3%, and an instruction to keep a level holds even when the user speaks differently. The remaining failures show that a model’s own history is a hazard. When an exhibitor leaves and the coordinator’s 22-year-old younger sibling starts asking questions, two models stay polite for all ten remaining replies, while two others also miss the change and recover on the next reply. In another dialogue, one reply that leaked internal markup was followed by 12 empty replies. Every failed listener-change reply came in a polite-to-casual change, and at 13 positions all five models chose 하십시오체 where reviewers expected 해요체. We release the scripts, replies, verdicts and code as a regression test. Files: main.pdf is the preprint (not peer reviewed). reproducibility.zip is the companion package: frozen scripts and expected labels, all 17,500 reply texts and saved verdicts, analysis code, and the manuscript source. Run python analysis/paper_diagnostics.py from the package root to reproduce the tables. Data are CC BY 4.0. Code in analysis/ is MIT.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24347926
Primary Topic
Speech and dialogue systems
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Polite to Whom? Benchmarking Korean Speech-Level Maintenance and Switching in Multi-Turn LLM Conversations

Prasanjeet Keshri
3 links
Zenodo (CERN European Organization for Nuclear Research)
Speech and dialogue systems
preprint

Polite to Whom? Benchmarking Korean Speech-Level Maintenance and Switching in Multi-Turn LLM Conversations

Prasanjeet Keshri
preprint en

Abstract

A Korean assistant chooses how to address its listener in every reply, and the right speech level can change mid-conversation. We introduce a benchmark that tests this choice across complete conversations that each model generates for itself. Its 250 fictional scripts cover keeping an assigned level, switching on request, switching when a new listener arrives, choosing a level from the relationship alone, and honorifying third parties. Five API models produced 17,500 replies, and three human reviewers verified a PASS/FAIL verdict for each. Mean reply success per dialogue is 97.8% to 99.3%, and an instruction to keep a level holds even when the user speaks differently. The remaining failures show that a model’s own history is a hazard. When an exhibitor leaves and the coordinator’s 22-year-old younger sibling starts asking questions, two models stay polite for all ten remaining replies, while two others also miss the change and recover on the next reply. In another dialogue, one reply that leaked internal markup was followed by 12 empty replies. Every failed listener-change reply came in a polite-to-casual change, and at 13 positions all five models chose 하십시오체 where reviewers expected 해요체. We release the scripts, replies, verdicts and code as a regression test. Files: main.pdf is the preprint (not peer reviewed). reproducibility.zip is the companion package: frozen scripts and expected labels, all 17,500 reply texts and saved verdicts, analysis code, and the manuscript source. Run python analysis/paper_diagnostics.py from the package root to reproduce the tables. Data are CC BY 4.0. Code in analysis/ is MIT.

Zenodo (CERN European Organization for Nuclear Research)
Speech and dialogue systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon