Knowledge distillation and dataset distillation of large language models: emerging trends, challenges, and future directions

The exponential growth of Large Language Models (LLMs) continues to highlight the need for efficient strategies to meet ever-expanding computational and data demands. This survey provides a comprehensive analysis of two complementary paradigms: Knowledge Distillation (KD) and Dataset Distillation (DD), both aimed at compressing LLMs while preserving their advanced reasoning capabilities and linguistic diversity. We first examine key methodologies in KD, such as task-specific alignment, rationale-based training, and multi-teacher frameworks, alongside DD techniques that synthesize compact, high-impact datasets through optimization-based gradient matching, latent space regularization, and generative synthesis. Building on these foundations, we explore how integrating KD and DD can produce more effective and scalable compression strategies. Together, these approaches address persistent challenges in model scalability, architectural heterogeneity, and the preservation of emergent LLM abilities. We further highlight applications across domains such as healthcare and education, where distillation enables efficient deployment without sacrificing performance. Despite substantial progress, open challenges remain in preserving emergent reasoning and linguistic diversity, enabling efficient adaptation to continually evolving teacher models and datasets, and establishing comprehensive evaluation protocols. By synthesizing methodological innovations, theoretical foundations, and practical insights, our survey charts a path toward sustainable, resource-efficient LLMs through the tighter integration of KD and DD principles.

Authors

Institutions

Publication Details

Journal
Artificial Intelligence Review
Published
2025-11-20
DOI
https://doi.org/10.1007/s10462-025-11423-3
Citations
9
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
5.89

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Knowledge distillation and dataset distillation of large language models: emerging trends, challenges, and future directions

Xianzu Gong, Wenqin Zhong, Xiaoming Zhaı, Tao Wang et al.
9 citations
Artificial Intelligence Review
Artificial Intelligence in Healthcare and Education
5.89
article

Knowledge distillation and dataset distillation of large language models: emerging trends, challenges, and future directions

Xianzu Gong, Wenqin Zhong, Xiaoming Zhaı, Tao Wang, Ehsan Latif, Dajiang Zhu, Jing Yan Cai, Zhengliang Liu, Xiaowei Yu, Soheil Kolouri, Shushan Wu, Ping Ma, Wei Ruan, Yongkai Chen, Wei Liu, Tianming Liu, Yufang Liu, Ali Abbasi, Wei Zhang, Jing Zhang, Luyang Fang, Terry Ma, Zhenyuan Yang, Haoran Lu
article en
9 citations

Abstract

The exponential growth of Large Language Models (LLMs) continues to highlight the need for efficient strategies to meet ever-expanding computational and data demands. This survey provides a comprehensive analysis of two complementary paradigms: Knowledge Distillation (KD) and Dataset Distillation (DD), both aimed at compressing LLMs while preserving their advanced reasoning capabilities and linguistic diversity. We first examine key methodologies in KD, such as task-specific alignment, rationale-based training, and multi-teacher frameworks, alongside DD techniques that synthesize compact, high-impact datasets through optimization-based gradient matching, latent space regularization, and generative synthesis. Building on these foundations, we explore how integrating KD and DD can produce more effective and scalable compression strategies. Together, these approaches address persistent challenges in model scalability, architectural heterogeneity, and the preservation of emergent LLM abilities. We further highlight applications across domains such as healthcare and education, where distillation enables efficient deployment without sacrificing performance. Despite substantial progress, open challenges remain in preserving emergent reasoning and linguistic diversity, enabling efficient adaptation to continually evolving teacher models and datasets, and establishing comprehensive evaluation protocols. By synthesizing methodological innovations, theoretical foundations, and practical insights, our survey charts a path toward sustainable, resource-efficient LLMs through the tighter integration of KD and DD principles.

Artificial Intelligence ReviewVol. 59(1)
Harvard University (US), University of Georgia (US), The University of Texas at Arlington (US), Vanderbilt University (US), Augusta University (US), Harvard University Press (US), Augusta University Health (US), Mayo Clinic Hospital (US), Mayo Clinic in Florida (US), Carnegie Mellon University (US)
National Science Foundation, National Institutes of Health, Institute of Education Sciences
Openalex Percentile: Top 4%
Artificial Intelligence in Healthcare and Education
5.89
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.