Selecting Pre-Training Philosophies for Large Language Models: A Case Study for Hindi NLP Tasks

The field of Natural Language Processing (NLP) has been revolutionized by the advent of Pre-trained Language Models (PLMs) based on transformer models. Despite the advantage provided by PLMs, NLP tasks on resource constrained languages such as Hindi perform poorly, primarily due to the lack of pre-training data. PLMs for such languages are built using three distinct methodologies - pre-training using an available monolingual training corpus, pre-training on a multilingual corpus, and fine-tuning a multilingual PLM on a monolingual corpus. All of these methodologies have been used in NLP tasks, but a comparative study among these methodologies have not been done in the context of resource constrained languages. In this work, we perform a comparative study of PLMs built using different design strategies for NLP tasks in Hindi. Experiments demonstrate that PLMs that were pre-trained on a multilingual corpus consisting of typologically similar languages outperform monolingual and multilingual PLMs. Furthermore, when these models are fine-tuned using a monolingual corpus, the performance further improves. Experiments also demonstrate that with a sufficiently large training corpus, monolingual PLMs outperform multilingual PLMs.