When More Metrics Do Not Improve Faithfulness Assessment: A Document-Level Evaluation of AI-Generated News Summaries
Automatic scores are attractive for screening artificial intelligence (AI)-generated news summaries, but their agreement with human judgments must be evaluated without sharing source articles between model fitting and testing. We examined whether a fixed linear fusion of five published scores improves the ranking of human-rated source faithfulness over the published entailment score alone. We reanalysed the Creative Commons licensed XSum hallucination-annotation release, using its existing human judgments and precomputed ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, and entailment scores. Before inspecting outcomes, we fixed mechanical eligibility rules, a deterministic document-level 80/20 split, ridge regression with penalty 1.0, Spearman correlation as the primary endpoint, and 2,000 document-cluster bootstrap draws. The eligible file contained 1,992 system-summary records from 498 BBC article IDs; 399 articles and 1,596 records were used for fitting, and 99 articles and 396 records for testing. On the held-out set, fusion correlated with human faithfulness at ρ = 0.425 (95% bootstrap interval 0.325 to 0.515), versus ρ = 0.445 (0.342 to 0.532) for entailment. The prespecified difference was 0.020 ( 0.100 to 0.066). For the secondary factuality-vote target, correlations were 0.285 and 0.285 after rounding, with no detectable fusion advantage. This reanalysis shows no evidence that this fixed fusion improves ranking over entailment in the released sample. The labels do not assess source attribution or editorial fidelity, and no new human annotation or summary generation was performed.
Authors
- Zhiwei Cheng (ORCID: https://orcid.org/0009-0004-1669-3275)
- Chaofan Yue
- Alina Andreeva
- Cheng Chen
- Jinjiang Liu
- Xiangyu Li
- Zyad Abdullah Al-Shuja
- Cuong Huynh
- Ramin Jalalpoor
Institutions
- Jiangxi University of Water Resources and Electric Power (CN)
- St Petersburg University (RU)
- ITMO University (RU)
- Shanghai Jiao Tong University (CN)
- Nanchang Institute of Technology (CN)
Publication Details
- Journal
- Journal of advances in engineering and technology.
- Published
- 2026-11-12
- DOI
- https://doi.org/10.62177/jaet.v3i4.1811
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 0.00