Efficiency Without Quality? A Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) Field Study of Sub-Intent Library Curation in a Live Airline Chatbot
This study asks whether the Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) cycle can improve utterance capture in a live enterprise chatbot, and how much of the change is attributable to the intervention during a concurrent migration to large language model (LLM) components. Turkish WhatsApp utterances of Turkish Airlines’ TK Assistant (80,295 utterances, January–July 2025) was analyzed; utterance capture scores (success threshold 0.95) were graded on a five-level scale, with 0.95 as the success threshold. After bidirectional Pareto analysis, 349 Strong-band utterances without a verbatim counterpart were added to ten sub-intent libraries. Assessment used utterance-level re-capture analysis against non-added controls, expected-versus-actual attribution, Laney p′ charts and blind human validation of 300 utterances. Recurring added texts were captured as exact matches (17 of 22 at first recurrence, against none of eight recurring controls; difference 77 points, 95% confidence interval 39–90), without organic promotion beyond near-identical forms. Under stated assumptions, the intervention was followed by 69 additional exact matches, while the successful utterance capture rate fell from 21.04% to 16.11%, a decline inseparable from the migration. Annotators judged exact matches correctly routed no more often than Strong-band matches (87% vs. 98%, p = 0.032 jointly, marginal per annotator). Verbatim curation buys exact-match efficiency, not understanding; confidence-based metrics should be secondary improvement targets.
Authors
- Mustafa Kaçan (ORCID: https://orcid.org/0009-0001-7957-0174)
- Selçuk Alp (ORCID: https://orcid.org/0000-0002-6545-4287)
Institutions
- Yıldız Technical University (TR)
Publication Details
- Journal
- Applied Sciences
- Published
- 2026-11-20
- DOI
- https://doi.org/10.3390/app162211506
- Primary Topic
- AI in Service Interactions
- Type
- article
- Field-Weighted Citation Impact
- 0.00