Benchmarking State-of-the-Art Text Summarization Models

Text summarization remains a crucial task in Natural Language Processing (NLP), with recent advancements in deep-learning models significantly improving performance. This paper presents a comprehensive review and evaluation of five innovative summarization models: BART, T5, PEGASUS, ProphetNet, and GPT-3. Study conducts extensive experiments on different summarization tasks using 1) content-based evaluation to measure the quality and coherence of generated summaries, 2) co-selection-based evaluation to measure human written summaries, and 3) document-based evaluation to measure content and organization of source documents. PEGASUS has the highest coverage and relevance when we performed co-selection-based evaluation. Content-based evaluation shows that GPT-3 performs better than others in all the parameters including BERTScore. Document based evaluation gives better content coverage, SummaQA, SUPERT with GPT-3 but divergence with T5. The comparative analysis provides valuable results into the current landscape of text summarization techniques.