Neural Machine Translation for Punjabi-English language pair using word-based tokenization

Machine Translation (MT) is becoming essential for communication, and Neural Machine Translation (NMT) is currently considered the most effective approach for language translation. However, limited research has been conducted on translation for low-resource Indian languages. Languages like Punjabi which are highly complex in terms of morphology and pose significant challenges during translation. As the usage of Indian language, especially Punjabi continue to grow, the requirement for its higher quality translation is increasing. The Transformer architecture based models have shown promising results in various NLP tasks for such low-resource languages. This paper proposes NMT based approach for Punjabi-English (PA-EN) and English-Punjabi (EN-PA) pair using the transformer architecture with word-based tokenization. The proposed work has been evaluated using publicly available Samanantar dataset. It outperforms the state-of-the-art by 3.5% for PA-EN and 3.64% for EN-PA as measured using BLEU score.