Domain-Specific Retrieval-Augmented Generation with Adaptive Embedding and Knowledge Distillation-Based Re-Ranking

Retrieval-augmented generation (RAG) has emerged as an effective approach for analyzing massive and diverse data. It offers promising avenues for energy management and intelligent decision support amid the accelerating digital transformation of the power industry. However, when applied to this specialized domain, traditional RAG systems face two key challenges: (1) poor comprehension of domain-specific terminology, leading to irrelevant retrieval, and (2) limited precision in re-ranking the retrieved results. To address these limitations, this paper presents an innovative integrated optimization framework. The framework enhances RAG performance in the electric power domain through two key strategies. First, we adapt a base embedding model to the domain using contrastive learning and iteratively refine hard negative samples to improve retrieval quality. Second, we employ a large language model (LLM) as a teacher to distill re-ranking knowledge into a lightweight bidirectional encoder representations from transformers (BERT) model, using a hybrid loss function that combines mean squared error (MSE) loss and margin ranking loss. The framework aims to simultaneously improve the model’s understanding of domain-specific terminology and the re-ranking accuracy of critical information. Experimental results on both a private power-domain dataset and the public DuReader_robust benchmark demonstrate that the proposed framework achieves significant performance gains. Comprehensive ablation studies confirm the necessity of each component and reveal their synergistic effects within the framework. Furthermore, sensitivity analyses of key hyperparameters confirm the effectiveness of our hybrid loss and identify optimal configurations that enhance both retrieval and generation performance. This work not only introduces an effective optimization framework tailored for domain-specific RAG applications but also advances industrial intelligence by enhancing the accuracy and reliability of information services.