Hierarchical Local-Global Transformer With Dynamic Positional Encoding for Document-Level Machine Translation

Existing document-level neural machine translation models aim to extend the translation unit from the sentence level to the entire document. However, current long sequence models face limitations due to the large hypothesis space problem associated with the standard attention mechanism. Moreover, as the length of the document increases, the translation performance decreases significantly. In this paper, we propose HLG, a <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><u>h</u></b>ierarchical <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><u>l</u></b>ocal-<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><u>g</u></b>lobal Transformer in a single-stream manner to address the issue of hypothetical space and we introduce a local bias into the positional encoding layer, mitigating the sparsity problem caused by absolute positional encoding. The hierarchical architecture with dynamic positional encoding can facilitate the interaction between the current sentence and the broader context to capture the feature in entire documents, which is beneficial for modeling long sequences. Experimental results demonstrate that the HLG achieves state-of-the-art translation performance in non-pre-training settings for document-level neural machine translation, outperforming various strong baselines in both sentence-to-sentence and document-to-document baseline manners. Furthermore, our model exhibits stable performance when extending the document modeling length up to longer tokens on three benchmark datasets.