Constructing a Multi-source Heterogeneous RAG Knowledge Q&A Dataset in the Telecom Domain

With the increasing complexity of the knowledge system in the telecom domain, existing telecom knowledge question answering (KQA) systems face numerous challenges, such as the lack of high-quality data and the difficulty in effectively evaluating the accuracy of knowledge question answering. To address these issues, this paper proposes the construction of a telecom knowledge Q&A dataset based on multi-source heterogeneous data fusion. The dataset is evaluated using various KQA strategies to assess its quality, aiming to improve the question answering generation performance in the telecom domain. Firstly, the research integrates open-source data, enterprise data, and desensitized project data to construct a high-quality dataset containing 6,296 telecom Q&A samples, named TeleRAG-6K. A hybrid quality control method, combining automatic labeling and manual review, is adopted to ensure the dataset’s high quality and adaptability. Secondly, a hybrid retrieval model based on BM25 and LlamaIndex is designed, significantly optimizing the cross-document information integration capability, and improving document recall and text block recall rates. Experimental results show that TeleRAG-6K performs well under DB-GPT, GRAFlow, and our proposed method. Moreover, compared to the DB-GPT and RAGFlow baseline models, our method achieves the best performance on telecom domain datasets, with a significant improvement in overall performance. The innovative approach in constructing the telecom RAG dataset, algorithm optimization, and evaluation methods proposed in this paper advance the application of RAG technology in the telecom vertical domain.