Entity hallucination index for text summarization models

Purpose Entity hallucination poses a critical issue in abstractive text summarization, where generated summaries include entities or groups not present in the original document. This study aims to systematically evaluate entity hallucination across state-of-the-art summarization models and address the gap in existing evaluation metrics by proposing a novel solution. Design/methodology/approach The research introduces the entity hallucination index (EHI) – a metric specifically designed to quantify the extent of entity hallucination in generated summaries. EHI categorizes hallucinations by their type and severity, enabling a fine-grained analysis of model outputs. The metric is applied to a range of leading summarization models to assess their faithfulness and hallucination tendencies. Findings The application of the EHI reveals notable variations in entity hallucination across different models. It effectively identifies and quantifies hallucinated entities, demonstrating its capability to differentiate models based on their fidelity to source content. These findings underscore the importance of hallucination-aware evaluation when comparing summarization models. Originality/value This article introduces a novel and focused evaluation metric – the EHI – that provides a much-needed quantitative framework for assessing entity-level hallucination. Unlike traditional metrics, EHI addresses a specific yet impactful dimension of summary quality, offering valuable insights for both researchers and practitioners involved in the development and deployment of summarization systems.