RAGVA: Engineering retrieval augmented generation-based virtual assistants in practice

Retrieval-augmented generation (RAG)-based applications are gaining prominence due to their ability to leverage large language models (LLMs). These systems excel at combining retrieval mechanisms with generative capabilities, resulting in contextually relevant responses that enhance user experience. In particular, Transurban, a road operation company, replaced its rule-based virtual assistant (VA) with a RAG-based VA (RAGVA) to offer flexible customer interactions and support a wider range of scenarios. This paper presents an experience report from Transurban’s engineering team on building and deploying a RAGVA, offering a step-by-step guide for creating a conversational application and engineering a RAGVA. The report serves as a reference for future researchers and practitioners. While the engineering processes for traditional software applications are well-established, the development and evaluation of RAG-based applications are still in their early stages, with numerous emerging challenges remaining uncharted. To address this gap, we conduct a focus group study with Transurban practitioners regarding developing and evaluating their RAGVA. We identified eight challenges encountered by the engineering team and proposed eight future directions that should be explored to advance the development of RAG-based applications. This study contributes to the foundational understanding of a RAG-based conversational application and the emerging AI software engineering challenges it presents.