Transformer-Based Model for High-Quality Text Summarization

The extraction of crucial information from large texts is making significant progress and is a necessity in all professions. In this era of social media, people seek quick and short bits of information. The use of deep learning and transformer models with encoder-decoder networks plays a huge role in abstractive text summarization. The transformer-based model proposed in this work leverages the transformer-based PEGASUS model for precise and human-like summarization. The model architecture is designed with an MLM (Masked Language Modelling) based encoder model, GSG (Gap Sentence Generation) based second masking tokenizer and a transformer decoder. Training this model involves fine-tuning GSG and optimizing the transformer-based decoder for precise text summarization. The utilization of GSG, MLM and the Transformer-based decoder marks a significant advancement in text summarization. The model was tested on diverse datasets, as demonstrated by its performance on both the SAMsum and CNN Daily mail datasets. The comprehensive result analysis highlighted the abstractive summarization results of the proposed model with ROUGE, BLEU and F1 values on the SAMsum dataset and CNN Daily mail dataset.