When More Metrics Do Not Improve Faithfulness Assessment: A Document-Level Evaluation of AI-Generated News Summaries

Automatic scores are attractive for screening artificial intelligence (AI)-generated news summaries, but their agreement with human judgments must be evaluated without sharing source articles between model fitting and testing. We examined whether a fixed linear fusion of five published scores improves the ranking of human-rated source faithfulness over the published entailment score alone. We reanalysed the Creative Commons licensed XSum hallucination-annotation release, using its existing human judgments and precomputed ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, and entailment scores. Before inspecting outcomes, we fixed mechanical eligibility rules, a deterministic document-level 80/20 split, ridge regression with penalty 1.0, Spearman correlation as the primary endpoint, and 2,000 document-cluster bootstrap draws. The eligible file contained 1,992 system-summary records from 498 BBC article IDs; 399 articles and 1,596 records were used for fitting, and 99 articles and 396 records for testing. On the held-out set, fusion correlated with human faithfulness at ρ = 0.425 (95% bootstrap interval 0.325 to 0.515), versus ρ = 0.445 (0.342 to 0.532) for entailment. The prespecified difference was 0.020 ( 0.100 to 0.066). For the secondary factuality-vote target, correlations were 0.285 and 0.285 after rounding, with no detectable fusion advantage. This reanalysis shows no evidence that this fixed fusion improves ranking over entailment in the released sample. The labels do not assess source attribution or editorial fidelity, and no new human annotation or summary generation was performed.

Authors

Institutions

Publication Details

Journal
Journal of advances in engineering and technology.
Published
2026-11-12
DOI
https://doi.org/10.62177/jaet.v3i4.1811
Primary Topic
Topic Modeling
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

When More Metrics Do Not Improve Faithfulness Assessment: A Document-Level Evaluation of AI-Generated News Summaries

Zhiwei Cheng, Chaofan Yue, Alina Andreeva, Cheng Chen et al.
Journal of advances in engineering and technology.
Topic Modeling
article

When More Metrics Do Not Improve Faithfulness Assessment: A Document-Level Evaluation of AI-Generated News Summaries

Zhiwei Cheng, Chaofan Yue, Alina Andreeva, Cheng Chen, Jinjiang Liu, Xiangyu Li, Zyad Abdullah Al-Shuja, Cuong Huynh, Ramin Jalalpoor
article en

Abstract

Automatic scores are attractive for screening artificial intelligence (AI)-generated news summaries, but their agreement with human judgments must be evaluated without sharing source articles between model fitting and testing. We examined whether a fixed linear fusion of five published scores improves the ranking of human-rated source faithfulness over the published entailment score alone. We reanalysed the Creative Commons licensed XSum hallucination-annotation release, using its existing human judgments and precomputed ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, and entailment scores. Before inspecting outcomes, we fixed mechanical eligibility rules, a deterministic document-level 80/20 split, ridge regression with penalty 1.0, Spearman correlation as the primary endpoint, and 2,000 document-cluster bootstrap draws. The eligible file contained 1,992 system-summary records from 498 BBC article IDs; 399 articles and 1,596 records were used for fitting, and 99 articles and 396 records for testing. On the held-out set, fusion correlated with human faithfulness at ρ = 0.425 (95% bootstrap interval 0.325 to 0.515), versus ρ = 0.445 (0.342 to 0.532) for entailment. The prespecified difference was 0.020 ( 0.100 to 0.066). For the secondary factuality-vote target, correlations were 0.285 and 0.285 after rounding, with no detectable fusion advantage. This reanalysis shows no evidence that this fixed fusion improves ranking over entailment in the released sample. The labels do not assess source attribution or editorial fidelity, and no new human annotation or summary generation was performed.

Journal of advances in engineering and technology.Vol. 3(4)
Jiangxi University of Water Resources and Electric Power (CN), St Petersburg University (RU), ITMO University (RU), Shanghai Jiao Tong University (CN), Nanchang Institute of Technology (CN)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon