Reverse Engineering of Chatbot Behaviour for Automated Testing

Abstract Conversational agents—or chatbots—are increasingly being used to access all sorts of services, like citizen services in city halls, customer support, or shopping. Moreover, recent advances in generative artificial intelligence are prompting the integration of conversational assistants into many applications, like programming IDEs, office automation software, or operating systems. Given the prominence of these agents, their correctness is a rising concern. However, automated and robust testing techniques for conversational systems are still needed. In this paper, we present a reverse engineering technique for extracting a behavioural model of a deployed chatbot (i.e., treated as a black-box) through the automated exploration of its functionality via Large Language Models. The extracted model is used for automated testing by generating testing conversation profiles, which a user simulator employs to conduct focused conversations with the chatbot-under-test. We describe our supporting tool Tracer , available as an integrated chatbot testing web platform at http://miso.ii.uam.es:8081/ . We report on three evaluations that show that our exploration technique can yield an accurate model of the chatbot-under-test (via coverage analysis), the subsequent testing can discover existing errors in the chatbot (via mutation testing), and the exploration is precise in discovering functionality in real-world chatbots in production built with heterogeneous technologies.