Implementing Generative AI as First-Line Support : Evaluating Fine-Tuning, Retrieval-Augmented Generation, and Prompt Engineering for Domain-Specific Chatbots

This thesis explores how generative AI can be implemented as first-line support in a small or medium-sized enterprise (SME), focusing on practical strategies for adapting large language models (LLMs) to company-specific needs. Conducted in collaboration with BM System, the study evaluates three adaptation methods: fine-tuning, Retrieval-Augmented Generation (RAG), and prompt engineering, using open-source models, with benchmarking against the closed-source GPT-4o-mini. Through a combination of automatic metrics (ROUGE), LLM-based assessments, and manual user surveys, the study finds that RAG, especially when paired with prompt engineering, provides the most scalable and effective solution for SMEs with limited technical resources. Fine-tuning enables domain adaptation but is constrained by data quality and hardware demands, while closed-source models outperform open-source alternatives but raise issues regarding cost, privacy, and control. The results lead to three implementation strategies, tailored to different organizational requirements. Importantly, the study emphasizes that creating value with AI in customer support is not only a technical challenge but also requires a critical understanding of the technology’s role and limitations, acknowledging that human expertise remains central to effective support. With a carefully chosen approach and critical reflection, generative AI can serve as a valuable complement to customer support in SMEs.