FedAK: a semi-supervised one-shot framework for heterogeneous federated learning via feature-level attention-based knowledge distillation

Federated learning (FL) enables multiple devices to collaboratively train a shared model while keeping their data localized, thereby preserving privacy. Despite its promise, FL continues to face key challenges such as model heterogeneity, high communication overhead, and performance degradation under non-independent and identically distributed (non-IID) data. This paper introduces FedAK, a semi-supervised one-shot FL framework that integrates feature-level attention with knowledge distillation to support efficient global learning while reducing data exposure. In FedAK, each client independently trains its local model in a fully supervised manner on private labeled data and transmits only the feature representations of a shared public dataset, greatly reducing communication costs. On the server side, a semi-supervised aggregation strategy is adopted, where an attention-based aggregation module is trained on a small labeled subset to generate informative ensemble features and soft pseudo-labels for a larger unlabeled subset. These pseudo-labels guide a global student model through knowledge distillation, eliminating the need for direct access to client data or logits. Extensive experiments across four benchmark datasets demonstrate that FedAK consistently outperforms four state-of-the-art one-shot FL methods, confirming its effectiveness under heterogeneous and non-IID settings.