Neuron-Level Language Tag Injection Improves Zero-Shot Translation Performance

Language tagging, a method whereby source and target inputs are prefixed with a unique language token, has become the de facto standard for conditioning Multilingual Neural Machine Translation (MNMT) models on specific language directions.This conditioning can manifest effective zero-shot translation abilities in MT models at scale for many languages.Expanding on previous work, we propose a novel method of language tagging for MNMT, injection, in which the embedded representation of a language token is concatenated to the input of every linear layer.We explore a variety of different tagging methods, with and without injection, showing that injection improves zeroshot translation performance with up to a 2+ BLEU score point gain for certain language directions in our dataset.