Mongolian-Chinese Cross-lingual Topic Detection Based on Knowledge Distillation

Most topic detection research focuses on cross-language information processing in languages with abundant resources, such as English-Chinese and English-German, while there are also some studies that focus on low-resource languages, for instance, Vietnamese-Chinese and Tibetan-Chinese cross-lingual topic detection. However, there is a scarcity of research on Mongolian-Chinese cross-lingual topic detection, mainly due to the scarcity of Mongolian language data and the limitations of traditional topic detection in Mongolian text representation, clustering, and topic representation. Therefore, this paper proposes a Mongolian-Chinese cross-lingual topic detection method based on knowledge distillation. First, the knowledge distillation method is utilized to integrate Mongolian-Chinese cross-language semantics, enabling Mongolian news to be mapped into the Chinese semantic space with rich semantics through a pre-trained Chinese language model. Then, during the fine-tuning process of the original model, a Parameter Efficient Fine Tuning (PEFT) module is added to prevent catastrophic forgetting of the pre-trained language model. Finally, a density-based hierarchical clustering algorithm is used to cluster the news texts. The experimental results show that the proposed method using the RoBERTa-WWM pre-trained model combined with the density-based hierarchical clustering algorithm has improved at least 6 percentage points in the F1 score and 2 percentage points in topic coherence evaluation metrics compared to other baseline models. Furthermore, after adding the PEFT module, the model has improved at least 4 percentage points in F1 score, 3 percentage points in topic coherence, and 3 percentage points in topic diversity, respectively. The experiments prove that this method can effectively improve the accuracy of Mongolian-Chinese cross-lingual topic detection.