Research on Low-Resource Neural Machine Translation Methods Based on Explicit Sparse Attention and Pre-trained Language Model

The development of machine translation is driven by the need for global communication through the automatic translation of words, sentences, and texts from one language to another. This paper proposes an improved Transformer-based model to enhance the performance of Kazakh-English machine translation and explores its feasibility and advantages in this field.The Transformer model, with its self-attention mechanism, has demonstrated state-of-the-art performance across various natural language processing tasks. While self-attention effectively captures long-range dependencies, it may extract irrelevant information from the context. To address this issue, we propose an algorithm called the Explicit Sparse Attention (ESA) algorithm. The sparse attention mechanism enhances focus on the global context by selecting the most relevant segments. In addition, a general form of attention enhancement for Transformer models is introduced, which allows for the integration of pre-trained language models and facilitates the modeling of word order relations between source and target languages.Experimental results, using publicly available datasets provided by the official bilingual websites of the Kazakhstan government (The Republic of Kazakhstan, 2023; Strategy, 2022; Primeminister, 2022), show that the proposed model achieves a BLEU score of 0.51, WER of 0.50, and TER of 0.43. The proposed model demonstrates significant advantages in the Kazakh-English low-resource translation field and achieves notable improvements over recent advancements in this domain.