Evaluating LLM-Based Chatbots through Touchpoint-Driven Process Models

This article introduces a method for evaluating the quality of large language model (LLM)-based chatbots, focusing on the user journey. Each dialogue utterance, whether from the user or the bot, is mapped to a predefined set of touchpoints that represent key interaction goals. The resulting dialogue sequences are treated as business process activities, enabling process mining to discover and analyze the process model underlying the dialogue context. This method identifies inefficiencies such as bottlenecks, conversational loops, and shifting responsibility to human agents. It was tested on simulated dialogues generated by an LLM-based financial chatbot, with user personas and dialogue goals also synthesized by LLMs. Although based on synthetic data, the results demonstrate the potential of process mining to uncover structural strengths and weaknesses in LLM-based chatbots.