Measuring the Summarization Capabilities of LLMs Using the ACT Score

Large Language Models (LLMs) have demonstrated significant capabilities in summarizing extensive textual content, yet their performance varies across different models. Evaluating the quality of these summaries is challenging due to the diverse aspects of summarization assessed by metrics such as ROUGE, BLEU, and BERTScore. This paper introduces an approach to measure summarization efficacy by examining the ability of LLM-generated summaries to answer questions from the ACT Reading section. We conducted experiments using GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet to generate summaries for ACT passages and assessed their accuracy in answering the associated questions. Our findings reveal that GPT-4o outperformed other models in most metrics and question-answering tasks, indicating its superior summarization capabilities. This method provides a novel perspective on evaluating summary quality, emphasizing practical application and comprehension.