A Tale of Evaluating Factual Consistency: Case Study on Long Document Summarization Evaluation

Ensuring factual consistency in summarization remains a challenge, especially for longdocument evaluation.While automated, reference-free evaluation models are essential given the impracticality of large-scale human assessment for lengthy texts, challenges persist in evaluating different systems on how to handle different summary granularities and evolving model generations.In this work, we conduct a systematic study on diverse factualconsistency evaluation systems across four long-document datasets, encompassing summaries generated by models from non-LLMs to proprietary LLMs.Our analysis reveals that fine-grained continuous scores can provide more reliable assessments of different evaluation systems' capabilities than binary classification.We also examine the relationship between sentence-level and summary-level model performance, highlighting its dependency on dataset characteristics.Moreover, our study reveals that advanced systems can achieve higher recall in error detection for older summaries, yet struggle with false positives and fine-grained error detection.Our analysis and case studies provide further insights into designing robust factuality evaluation systems, which are becoming increasingly in demand as generative models advance rapidly.Ex 1: They were all men who paid for their ship with their lives.Label: SentE; Generator: Phi-2; All systems failed to identify the error.