Joint Class and Domain Continual Learning for Decentralized Federated Processes

Continual learning (CL) is attracting attention to learn in dynamic and non-stationary environments, typical of many real applications, such as those related to Internet of things (IoT). So far, learning in IoT scenarios has almost exclusively been studied in a centralized fashion, but CL can be used in decentralized settings involving a multitude of nodes to learn, adapt, and generalize from their streams of data, even with limited communication and computation capabilities. In this work, we formulate a joint Class and Domain Continual Learning problem, where local data at the nodes might include multiple classes belonging to different domains. In our solution, we adopt knowledge distillation (KD) to share knowledge extracted from local models trained with the data at each node and to incrementally generate a global model, while preserving privacy and avoiding centralization bottlenecks. Thanks to KD, the learning process can be performed without any prior information about the model architecture, i.e., every local model can be based on different architectures, which can also differ from the global. The learning process is also agnostic to domain and classes’ distribution across the nodes. With these features, our distributed learning solution may assist in privacy preservation and save communication resources, which are fundamental challenges in IoT scenarios. Compared against two baseline losses based on state-of-the-art CL solutions and using different datasets (MNIST, USPS, DomainNet), our proposal outperforms different baselines in terms of accuracy, spanning from a gain of 15.9%, up to 95.4%, by limiting catastrophic forgetting.