Automated end-to-end testing for conversational agents

The advances in generative artificial intelligence, especially Large Language Models (LLMs), have prompted the proliferation of conversational agents (or chatbots). These can be general-purpose – like ChatGPT – or tailored to specific tasks – like buying tickets or obtaining customer support. Although chatbots play a significant role in today’s software ecosystem, they are hard to test: defining meaningful, thorough tests is time-consuming, and setting an oracle flexible to conversational variations is challenging. This is aggravated when testing LLM-based chatbots, as their conversation is natural but unpredictable. To alleviate this problem, we present an end-to-end testing approach for conversational agents, comprising two components. First, a highly customisable user simulator that generates meaningful conversations with a chatbot under test, for the given goals (e.g., setting an appointment) and communication styles (e.g., long/short phrases, spelling mistakes). Second, a domain-specific language to specify and check correctness conditions (assertions and metamorphic relations) on the generated conversations. The conditions can assess functional correctness (e.g., booking more tickets costs more) and interaction styles (e.g., the chatbot responds in English and does not deviate from certain topics). This paper describes the approach, an implementation enabling chatbots’ testing independently of their technology, and an evaluation of its effectiveness in finding defects. We tested our tool on chatbots with artificially injected errors, and on third-party, real-world chatbots. Our tool detected between 81.25% and 100% of the injected errors, and identified actual functional issues in the real-world chatbots by applying manually defined correctness rules.