Evaluating Open-Source LLMs in RAG Systems: A Benchmark on Diploma Theses Abstracts Using Ragas

Abstract Retrieval-Augmented Generation (RAG) systems have gained significant attention for their ability to enhance language models by incorporating external knowledge sources. However, evaluating the effectiveness of these systems remains a challenge, as both the retrieval and generation components must be assessed independently and in combination. This study aims to provide a comprehensive evaluation of open-source large language models (LLMs) within a RAG system, using a dataset derived from diploma theses abstracts. To systematically assess performance, we generated a benchmark dataset consisting of 122 questions with reference answers, categorized into reasoning, fact-based, and summary-type questions. Using Elasticsearch as the retriever and several open-source LLMs as generators, we evaluated the system using the Ragas (Retrieval Augmented Generation Assessment) framework, focusing on retrieval effectiveness and answer quality. Our findings highlight the strengths and weaknesses of different models and retrieval strategies, offering insights into optimizing RAG implementations for academic and structured knowledge tasks. Our dataset and the corresponding evaluation code are publicly accessible at https://github.com/margitantal68/rag_paper .