Evaluating the Effectiveness of a Rule-Based WhatsApp Chatbot in Automated Query Resolution

Conversational chatbots have become integral to automated query resolution across industries to enhance customer service efficiency. However, their effectiveness in accurately addressing diverse customer queries remains a critical area of evaluation. This study examines the performance of a rule-based WhatsApp chatbot deployed by a South African telecommunications service provider, focusing on its ability to resolve customer queries effectively. This paper analysed chatbot interaction reports and quantifies success, failure, and abandonment rates across various query types. This study adopted a quantitative research approach with interpretivism as a philosophical paradigm. Furthermore, document analysis was employed for the analysis of TELCO X WhatsApp Chat’s extracted reports to examine and evaluate human-chatbot interaction. This method allowed the researcher to determine, from the interactions, resolved and unresolved queries. The findings indicate that the chatbot achieves high success rates for routine and structured queries, such as Frequently Asked Questions (FAQs) and retrieving account-related information. However, the chatbot’s performance declines significantly in handling complex, multi-step, or context-dependent queries, including SIM swaps and product purchases. The chatbot exhibits lower success rates, with a significant number of customer interactions resulting in unresolved queries or user abandonment. The analysis highlights key performance limitations, including natural language understanding, contextual retention, and the inability to process multi-step interactions effectively. Additionally, the absence of seamless escalation mechanisms to human agents contributes to customer frustration when the chatbot fails to provide satisfactory resolutions. The study provides recommendations for optimising chatbot performance, particularly in enhancing AI-driven response mechanisms, refining intent recognition, and using advanced dialogue management techniques. Additionally, integrating seamless escalation pathways to human agents is proposed to improve resolution rates for complex queries. In conclusion, the study emphasizes the importance of continuous performance monitoring and iterative improvements. Identifying key performance gaps and improvement areas provides valuable guidance for organisations looking to optimise chatbot functionality.