Heterogeneous knowledge distillation via architecture-aware reinforcement learning

Knowledge distillation (KD) transfers knowledge from a large teacher network to a compact student network. A central challenge in heterogeneous multi-teacher KD is balancing supervision from teachers with fundamentally different architectures—convolutional neural networks (CNNs), vision transformers (ViTs), and MLP-Mixers—whose features occupy incompatible representation spaces. Most existing methods either assume homogeneous teachers or use uniform weighting, without jointly addressing architecture-adaptive alignment and dynamic teacher weighting. We propose Heterogeneous Knowledge Distillation via Architecture-Aware Reinforcement Learning (HARKD), a framework that combines three components: (1) architecture-adaptive projectors that map heterogeneous features into compatible alignment spaces; (2) a heterogeneity-aware state representation that captures teacher performance, architectural compatibility, and branch-level alignment quality; and (3) reinforcement-learning-based optimization that dynamically weights teacher contributions across multiple student exits. The learned policy assigns depth-dependent teacher weights as the student evolves during training. Experiments on ImageNet-1K, CIFAR-100, and Tiny-ImageNet show that HARKD consistently outperforms the evaluated multi-teacher KD baselines.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-14
DOI
https://doi.org/10.1038/s41598-026-76383-2
Primary Topic
Advanced Neural Network Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Heterogeneous knowledge distillation via architecture-aware reinforcement learning

Zeng Qia, Huilong Fan, Yibo Yang
Scientific Reports
Advanced Neural Network Applications
article

Heterogeneous knowledge distillation via architecture-aware reinforcement learning

Zeng Qia, Huilong Fan, Yibo Yang
article en

Abstract

Knowledge distillation (KD) transfers knowledge from a large teacher network to a compact student network. A central challenge in heterogeneous multi-teacher KD is balancing supervision from teachers with fundamentally different architectures—convolutional neural networks (CNNs), vision transformers (ViTs), and MLP-Mixers—whose features occupy incompatible representation spaces. Most existing methods either assume homogeneous teachers or use uniform weighting, without jointly addressing architecture-adaptive alignment and dynamic teacher weighting. We propose Heterogeneous Knowledge Distillation via Architecture-Aware Reinforcement Learning (HARKD), a framework that combines three components: (1) architecture-adaptive projectors that map heterogeneous features into compatible alignment spaces; (2) a heterogeneity-aware state representation that captures teacher performance, architectural compatibility, and branch-level alignment quality; and (3) reinforcement-learning-based optimization that dynamically weights teacher contributions across multiple student exits. The learned policy assigns depth-dependent teacher weights as the student evolves during training. Experiments on ImageNet-1K, CIFAR-100, and Tiny-ImageNet show that HARKD consistently outperforms the evaluated multi-teacher KD baselines.

Scientific Reports
University of Electronic Science and Technology of China (CN)
Advanced Neural Network Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon