Optimizing RAG in Programming Education: A Comparative Study of Models and Strategies

This study examines how Retrieval-Augmented Generation (RAG) enhances the effectiveness of generative artificial intelligence in programming education. This study compares commercial and open-source large language models within a RAG system, examining how retrieval design and prompt engineering affect response quality. By creating a database specific to machine learning courses and using the RAGAS (Retrieval-Augmented Generation Assessment) evaluation framework, a comparative analysis of five prominent models (GPT-4o, Claude-3.7-Sonnet, Gemini-2.0-Flash, Llama3.3-70b, and Ministral-8b) is conducted based on five quality metrics, namely Context Precision, Context Recall, Context Entities Recall, Faithfulness, and Response Relevance. The findings reveal considerable performance differences between different model types, with structured reasoning prompts (Chain-of-Thought and Take a Step Back) significantly improved the overall model performance. Re-ranking was identified as the most effective retrieval approach, especially in enhancing the performance of lightweight open-source models. This research provides empirical evidence for the effectiveness of economically feasible RAG systems in programming education, thus helping bridge the gap in the performance of commercial and open-source models and providing real-world solutions for resource-constrained educational settings.