Assessing the Reliability and Validity of the Measures for Automatic Text Summarization

Automatic Text Summarization (ATS) is a research area that originated in the late 1950s and has gained increasing importance with the surging amount of text data available today. One of the key challenges in this area is how to quantitatively assess the quality of the summaries produced. The three most widely quantitative measures used for this task are: ROUGE, BLEU and BERTScore. This paper attempts to comparatively evaluate the validity and reliability of such measures. The concept of Shannon' entropy from information theory served as background for this work. Experiments were conducted using the CNN corpus, focusing on news articles written in English.