Heterogeneous knowledge distillation via architecture-aware reinforcement learning
Knowledge distillation (KD) transfers knowledge from a large teacher network to a compact student network. A central challenge in heterogeneous multi-teacher KD is balancing supervision from teachers with fundamentally different architectures—convolutional neural networks (CNNs), vision transformers (ViTs), and MLP-Mixers—whose features occupy incompatible representation spaces. Most existing methods either assume homogeneous teachers or use uniform weighting, without jointly addressing architecture-adaptive alignment and dynamic teacher weighting. We propose Heterogeneous Knowledge Distillation via Architecture-Aware Reinforcement Learning (HARKD), a framework that combines three components: (1) architecture-adaptive projectors that map heterogeneous features into compatible alignment spaces; (2) a heterogeneity-aware state representation that captures teacher performance, architectural compatibility, and branch-level alignment quality; and (3) reinforcement-learning-based optimization that dynamically weights teacher contributions across multiple student exits. The learned policy assigns depth-dependent teacher weights as the student evolves during training. Experiments on ImageNet-1K, CIFAR-100, and Tiny-ImageNet show that HARKD consistently outperforms the evaluated multi-teacher KD baselines.
Authors
- Zeng Qia
- Huilong Fan (ORCID: https://orcid.org/0000-0003-4350-9299)
- Yibo Yang
Institutions
- University of Electronic Science and Technology of China (CN)
Publication Details
- Journal
- Scientific Reports
- Published
- 2026-11-14
- DOI
- https://doi.org/10.1038/s41598-026-76383-2
- Primary Topic
- Advanced Neural Network Applications
- Type
- article
- Field-Weighted Citation Impact
- 0.00