Pre-ordering representations improve low-resource neural machine translation and application in the Māori language

Abstract Pre-ordering is a data pre-processing technique that reorganizes the word order in a source language sentence to better align with the syntax of the target language, and it has been shown to significantly impact various tasks. While previous studies have primarily leveraged token position embeddings in pre-ordered sentences to enhance machine translation, they have not directly explored learning contextualized representations that encapsulate richer semantic information and also reflect the word position information in the pre-ordered sentences. In this work, we introduce a novel pre-ordering-aware neural network that explicitly integrates the representations of pre-ordered sentences into the training process. Our network utilizes a pre-ordering encoder, which works alongside the original Transformer encoder to process the pre-ordered source sentence. We propose a Cross-Encoder Consistency (CEC) block, which encourages the original encoder to produce representations that better reflect the word order of the target language by closing the gap between representations of the original and pre-ordered sentence. Additionally, we incorporate a Sentence Encoding Consistency (SEC) block to help the model preserve the semantic integrity of the source sentence. We conduct extensive experiments on multiple low-resource neural machine translation benchmarks, including an endangered language, the Māori language. Our method delivers substantial enhancements in translation quality, achieving improvements of up to +1.46 BLEU in Tr $$\rightarrow$$ En translation.