MetaCo-Distillation: A Robust Collaborative Distillation Framework for Multivariate Knowledge

Federated Learning (FL), as a rapidly evolving privacy-preserving collaborative machine learning paradigm, has shown great potential in the emerging Metaverse areas. Although numerous studies have theoretically demonstrated the feasibility of FL, practical applications in the Metaverse face challenges such as learning forgetting due to non-independent and identically distributed (non-i.i.d.) data, high communication overhead from model parameter transmission, and the inability to accommodate heterogeneous models. To address these challenges, this paper proposes a collaborative distillation framework called Co-distillation. In this distributed environment, each device independently trains its local model using private data, communicates predictions on an unlabeled public dataset, and generates surrogate labels for the public dataset based on received predictions. These surrogate labels, along with the received predictions, serve as teacher logits for local model distillation. This approach is more adaptable to heterogeneous streaming data continuously generated by industrial devices, offers higher communication efficiency than traditional methods, allow for heterogeneous models. Experiments have been conducted, demonstrating the superior performance of the proposed framework.