Text Summary Generation based on Data Augmentation and Contrastive Learning

In recent years, there has been a rapid growth in the volume of textual data generated from various sources, including industries, news media, and social media, across various fields worldwide. It contains valuable information and knowledge, but its sheer volume requires effective summarization techniques to make it useful. Text summarization has thus become an important technology for distilling large amounts of information and capturing key insights. With advancements in large language models (LLMs), most existing abstractive summarization models leverage pretrained models such as BART. These models are based on Transformer architectures and undergo unsupervised pretraining, followed by fine-tuning on specific datasets for downstream tasks, particularly text summarization. These models have demonstrated superiority in this field. Real-world data often contains noise, and most pretrained language models suffer from exposure bias and a discrepancy between their training objectives and real evaluation metrics. In this paper, we introduce sentence-level data augmentation to simulate real-world text variations and enhance the model's denoising capability. Furthermore, contrastive learning is employed to learn to distinguish the differences between the original article with candidate summaries and among candidate summaries themselves. Finally, the highest-scoring article is output as the resulting summary. The experimental results on the CNN/DailyMail and XSum datasets indicate that this model shows improvement in generating text summaries compared to pretrained models and existing methods with contrastive learning. When comparing to the multitask learning framework BRIO variant with contrastive loss, our proposed method achieves a better ROUGE-1 of 48.52 and comparable ROUGE-2/L of 24.84 and 39.78. This shows the potential of our proposed method. Further investigation is needed to verify the performance in real-world texts in various domains.