Efficiency Without Quality? A Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) Field Study of Sub-Intent Library Curation in a Live Airline Chatbot

This study asks whether the Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) cycle can improve utterance capture in a live enterprise chatbot, and how much of the change is attributable to the intervention during a concurrent migration to large language model (LLM) components. Turkish WhatsApp utterances of Turkish Airlines’ TK Assistant (80,295 utterances, January–July 2025) was analyzed; utterance capture scores (success threshold 0.95) were graded on a five-level scale, with 0.95 as the success threshold. After bidirectional Pareto analysis, 349 Strong-band utterances without a verbatim counterpart were added to ten sub-intent libraries. Assessment used utterance-level re-capture analysis against non-added controls, expected-versus-actual attribution, Laney p′ charts and blind human validation of 300 utterances. Recurring added texts were captured as exact matches (17 of 22 at first recurrence, against none of eight recurring controls; difference 77 points, 95% confidence interval 39–90), without organic promotion beyond near-identical forms. Under stated assumptions, the intervention was followed by 69 additional exact matches, while the successful utterance capture rate fell from 21.04% to 16.11%, a decline inseparable from the migration. Annotators judged exact matches correctly routed no more often than Strong-band matches (87% vs. 98%, p = 0.032 jointly, marginal per annotator). Verbatim curation buys exact-match efficiency, not understanding; confidence-based metrics should be secondary improvement targets.

Authors

Institutions

Publication Details

Journal
Applied Sciences
Published
2026-11-20
DOI
https://doi.org/10.3390/app162211506
Primary Topic
AI in Service Interactions
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Efficiency Without Quality? A Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) Field Study of Sub-Intent Library Curation in a Live Airline Chatbot

Mustafa Kaçan, Selçuk Alp
1 links
Applied Sciences
AI in Service Interactions
article

Efficiency Without Quality? A Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) Field Study of Sub-Intent Library Curation in a Live Airline Chatbot

Mustafa Kaçan, Selçuk Alp
article en

Abstract

This study asks whether the Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) cycle can improve utterance capture in a live enterprise chatbot, and how much of the change is attributable to the intervention during a concurrent migration to large language model (LLM) components. Turkish WhatsApp utterances of Turkish Airlines’ TK Assistant (80,295 utterances, January–July 2025) was analyzed; utterance capture scores (success threshold 0.95) were graded on a five-level scale, with 0.95 as the success threshold. After bidirectional Pareto analysis, 349 Strong-band utterances without a verbatim counterpart were added to ten sub-intent libraries. Assessment used utterance-level re-capture analysis against non-added controls, expected-versus-actual attribution, Laney p′ charts and blind human validation of 300 utterances. Recurring added texts were captured as exact matches (17 of 22 at first recurrence, against none of eight recurring controls; difference 77 points, 95% confidence interval 39–90), without organic promotion beyond near-identical forms. Under stated assumptions, the intervention was followed by 69 additional exact matches, while the successful utterance capture rate fell from 21.04% to 16.11%, a decline inseparable from the migration. Annotators judged exact matches correctly routed no more often than Strong-band matches (87% vs. 98%, p = 0.032 jointly, marginal per annotator). Verbatim curation buys exact-match efficiency, not understanding; confidence-based metrics should be secondary improvement targets.

Applied SciencesVol. 16(22)
Yıldız Technical University (TR)
AI in Service Interactions
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon