FedHDE: an efficient federated learning model based on hyper-knowledge distillation

Abstract The heterogeneity of data between different clients limits the performance of federated learning training, and hyper-knowledge distillation has emerged as a potential solution to address heterogeneous data. However, existing hyper-knowledge distillation methods often sacrifice efficiency to improve model performance. To address these challenges, this paper proposes FedHDE, a novel federated learning framework based on hyper-knowledge distillation and efficient tensor optimization. FedHDE integrates a Dynamic Client Selection strategy within a teacher–assistant–student hierarchical structure to enhance global aggregation accuracy and reliability without relying on public datasets or server-side models. Furthermore, an efficient tensor optimization algorithm is introduced to improve numerical stability and reduce computation time. Theoretical analysis demonstrates that FedHDE achieves a convergence rate of $$O(1/\sqrt{T})$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>O</mml:mi> <mml:mo>(</mml:mo> <mml:mn>1</mml:mn> <mml:mo>/</mml:mo> <mml:msqrt> <mml:mi>T</mml:mi> </mml:msqrt> <mml:mo>)</mml:mo> </mml:mrow> </mml:math> , ensuring efficient and stable learning. Experimental results on the CIFAR10 dataset show that FedHDE improves both local and global accuracy by 0.14%–4.32%, while reducing training time by 65.28% compared to state-of-the-art baselines.Code is public at https://anonymous.4open.science/r/FedHDE .