EADSum: Element-Aware Distillation for Enhancing Low-Resource Abstractive Summarization

Abstractive summarization aims to generate concise summaries from input documents, with notable advancements driven by the advent of large language models (LLMs). However, the substantial computational demands and the sheer size of LLMs pose significant scalability challenges for the corresponding deployment in real-world applications. Existing methods typically adopt large amounts of training data to train smaller, more compact models to achieve performance on par with LLMs, either through fine-tuning with human-annotated labels or distilling rationales from LLM-generated labels. As an appealing alternative for low-resource abstractive text summarization, we propose EADSum (Element-Aware Distillation for Summarization), a novel training framework which aims to generate fine-grained summaries correlated to the human writing mindset while alleviating the heavy requirement of supervised training data. The proposed EADSum approach first guides LLMs to generate element-aware rationales from the input document, drawing attention to crucial elements such as entities, dates, events, and the results of event. These generated rationales then serve as additional supervision for the subsequent training of compact models within a multi-task learning framework. A series of experiments conducted on the CNN/DailyMail benchmark dataset demonstrate the feasibility and effectiveness of our approach.<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>