Abstractive Summarization of Khasi Texts using Pretrained Large Language Models

An automatic abstractive summarization model for the Khasi news articles is implemented in this work by leveraging language models that have emerged at the forefront in the recent past. The emergence of language models has opened avenues for the exploration and implementation of abstractive summarization models even in low-resource languages because the approach may be applied even without the support of other linguistic tools as it mainly relies on the availability of a sizable corpus. Since the Khasi language lacks linguistic tools and resources, this is a practical and effective approach. The methodology relies on the availability of two large language models pre-trained on generic texts for natural language understanding. Thus, a RoBERTa (Robustly Optimized BERT Pretraining Approach) model pre-trained in the Khasi language on a generic corpus of 400 mb Khasi texts and a publicly available pre-trained GPT-2 (Generative pre-trained transformer) has been utilized. For implementation, an encoder-decoder network has been employed for this sequence-to-sequence task. The pre-trained RoBERTa is loaded into the encoder since RoBERTa is an encoder-only model while the decoder is loaded with a pre-trained GPT-2 as the GPT-2 is a decoder-only model. The encoder transforms the input sequence to contextualized embeddings and during decoding these contextualized outputs from the encoder are used by the decoder for calculating the cross-attention between the inputs and the target summary sequence. The decoder outputs a probability distribution of the words that form the summary. For training and finetuning the summarization model, a corresponding summarization corpus has to be constructed as there is no such corpus available publicly. Since the approach requires a sizable corpus, a relatively large dataset must be built and updated incrementally. The model achieves ROUGE-1 scores of 0.55, 0.29, and 0.37 for recall, precision, and F1 respectively, ROUGE-2 scores of 0.15, 0.11, and 0.12 for recall, precision, and F1 respectively, and ROUGE-L scores of 0.14, 0.12 and 0.12 for recall, precision, and F1 respectively. These are good performance scores considering a relatively small corpus size of 10906 article-summary pairs.