Text Summarization with LLM: A Comparison of Transformer and Non-Transformer Models

Natural language processing, also known as NLP, relies heavily on assessing the quality of generated text, such as machine translations, summaries, and captions. Traditional assessment measures, such as ROUGE and BLEU, rely heavily on surface-level n-gram matching, which frequently fails to capture semantic complexity, paraphrasing, and synonymy. While these metrics perform well in structured text, they struggle to appropriately evaluate more complicated language generating tasks involving context, semantics, and linguistic fluency. This constraint emphasizes the necessity for a more sophisticated evaluation approach that is closely aligned with human judgment. Recent advances in NLP, notably transformer models like BERT and BART, enable the collection of more detailed contextual language representations. BERTSCORE, a novel metric based on these models, calculates text similarity by comparing token embeddings in candidate and reference sentences. This contextualized method enables a more complex evaluation that goes beyond exact word matching and corresponds well with human judgment in a variety of tasks, including automated translation and summarizing. A crucial problem, however, is establishing a metric that not only outperforms existing approaches but also achieves consistency over a wide range of text kinds and languages. Exploring BERTSCORE's applicability to additional pre-trained models and domain-specific applications could boost its effectiveness, fulfilling the need for a robust, computationally efficient, and semantically accurate evaluation standard.