Enhancing Federated Knowledge Distillation in Heterogeneous and Non-IID Scenarios

Federated Learning (FL) allows multiple participants to train models together while keeping their data private. Some FL frameworks use Knowledge Distillation to address model heterogenity, but many struggle in non-IID and heterogeneous environments, making it hard for clients to learn from each other. In this work, we show that the entropy of the softmax-averaged logits from clients reflects the model’s convergence. Based on this, we propose a new loss function, Sharpened Symmetric KL Divergence Loss (SSKL), which combines KL and Reverse KL Divergence with Label Sharpening to reduce the impact of non-IID data. Experiments demonstrate that our approach improves performance and reduces accuracy decline in non-IID and heterogeneous settings.