Character-Level Encoding based Neural Machine Translation for Hindi language

Neural Machine Translation (NMT) is one step ahead of traditional statistical phrase-based translation systems because of its better translation ability. But it requires a large amount of parallel training data, which can be challenging for languages with limited resources like many Indian languages. In the past, researchers have tried to address the issue using data augmentation. In this paper, we present a data augmentation technique for the Hindi language based on five phrases: noun phrases, verb phrases, prepositional phrases, adjective phrases, and adverb phrases. We augment the training corpus using parser-generated phrasal segments and evaluate the efficiency of the proposed work on the Hindi language. Further, the paper presents training in the NMT model at the character level instead of the word level. This approach can help overcome challenges associated with word-level translations, such as handling rare and out-of-vocabulary words and phrases, dealing with morphological complexity, and addressing languages with ambiguous word boundaries. The proposed work was evaluated on a low-resource language pair, Hindi-English, using the Google Transformer model as the baseline state-of-the-art. The experiments used two distinct datasets: WMT14 Hin-Eng and Samanantar Hin-Eng parallel corpus with character-level encoding for the translation task. The proposed model is able to surpass the cutting-edge baseline and saw an increase in BLEU scores for the WMT14 translation challenge with +2.52 on base paper using three phrase sentences with character-level encoding and +2.68 BLEU Score on base paper using five phrase sentences with character-level encoding. Further, character-level encoding is evaluated on non-augmented Samanantar dataset; it performs better in the baseline approach for translation purposes. It clearly shows that the proposed model outperforms in Hindi language translation.