Advancing Text Summarization with Knowledge Distillation: Techniques, Datasets, and Challenges

Text summarization (TS) is a crucial task in Natural Language Processing (NLP), enabling the transformation of lengthy content into precise versions while preserving essential meaning. However, recent advanced summarization approaches are highly computational costly, thus, further limit their usability in resource-constrained environments. Knowledge Distillation (KD) addresses this challenge by allowing smaller student models to learn from larger teacher models, reducing resource requirements while maintaining efficiency. This survey presents a comprehensive review of KD techniques for abstractive text summarization, categorizing methods based on objectives, distillation strategies, and teacher-student architectures. Additionally, it examines datasets used in KD-based summarization and evaluates performance trade-offs, strengths, and limitations across benchmarks. Finally, the current study discusses trends, challenges, and future research directions, emphasizing domain adaptability, improved alignment strategies, and leveraging large language models as teachers to enhance KD frameworks in text summarization.