A Multi-Objective Alternating Knowledge Distillation Method for Heterogeneous Federated Learning

Federated learning has shown great potential in protecting client privacy during large-scale model training. However, the challenge of non-IID data in Industrial Internet of Things environments makes it difficult to balance between preventing catastrophic forgetting and adapting to new tasks during model training. This paper proposes a Federated Learning algorithm via multi-objective optimization and continual learning (FedMoCL). Specifically, a novel federated alternating learning (FedAL) Framework is designed, which alternately optimizes plasticity and stability in the Continual Learning process through two distinct knowledge distillation modes: Teacher–Student Knowledge Distillation from the current server to the current client and self-knowledge distillation from historical clients to the current client. Furthermore, a multi-objective embedded optimization (MEO) is designed for FedAL, where sampled clients are considered as the evolutionary population and the entire set of clients as the decision space. Before federated communication, the plasticity-stability trade-off Pareto Solution is selected for aggregation. Simulation results demonstrate that FedMoCL effectively alleviates the data heterogeneity problem in federated foundation model training.