Corpus Quality Assessment Algorithm Driven by Pre-Trained Language Models

To solve the problems of low efficiency and single dimension of traditional corpus quality evaluation methods, this paper proposes a quality evaluation algorithm based on BERT to conduct a comprehensive evaluation of the corpus from five dimensions: grammatical correctness, semantic coherence, fact consistency, domain adaptability and diversity. In the evaluation of grammatical correctness, the powerful context encoding ability of BERT is utilized to capture the grammatical structure. Semantic coherence assessment determines coherence by analyzing the semantic relations of the text. The fact consistency assessment combines the reasoning capabilities of KG and BERT to verify the accuracy of facts. Domain adaptability assessment is judged based on domain feature matching and the generalization ability of BERT. Diversity assessment explores the differences in corpora from multiple levels. The results show, the algorithm has shown certain effectiveness in various evaluation tasks, but there is still room for improvement in complex semantic understanding and knowledge reasoning.