When the Reference Is Extracted: TextRank Outscores Fine-Tuned Transformers and a Small LLM on BBC News Summary

BBC News Summary is a small, widely used dataset for teaching and comparing text summarisers, yet how its reference summaries were made is not documented. We audit the dataset and then compare fourteen systems on one leakage-free split: four extractive baselines, three zero-shot models, six fine-tuned encoder-decoder models and a 1.5B-parameter instruction-tuned language model adapted with LoRA, with three seeds per fine-tuned model. Of the 2,225 articles, 154 are near-duplicates; under a naive random split, 11.5% of test articles would have a copy in training. 94.8% of reference sentences occur verbatim in the article, and a reference is on average 45% as long as its article. On 311 held-out articles, TextRank given the training-set length ratio scores 82.9 ROUGE-Lsum and 77.3 ROUGE-2, ahead of the best fine-tuned model, PEGASUS-large (72.7 and 64.5; paired difference +10.2, 95% CI 8.7 to 11.5), and higher on 81% of articles; the gap persists on BERTScore. Every fine-tuned model beats a length-matched Lead baseline on ROUGE-2, but Flan-T5-base and Flan-T5-large do not on ROUGE-Lsum, and Flan-T5-large, which writes the shortest summaries, scores below its smaller sibling. Zero-shot models score 25 to 38 ROUGE-Lsum, mostly because of length and wording, and the zero-shot LLM writes numbers absent from the article (7.8% of its numbers, 0.8% after LoRA). We conclude that BBC News Summary mainly measures how well a system reproduces a sentence-selection rule, and that results on it should always be reported next to a length-matched TextRank baseline. Code, the fixed split, per-document scores and the Kaggle notebook: https://github.com/MHKhan50/NewsSum-Bench (software archive: https://doi.org/10.5281/zenodo.24046039). Models and browser demo: https://huggingface.co/MHkhan50