Temperature-Aware Adaptive Federated Distillation for Energy-Constrained AIoT with Non-IID Data

Federated Learning (FL) can help multiple Internet of Things (IoT) devices to collaboratively train a machine learning model to provide intelligent services and applications (FL-AIoT). Due to IoT devices' limited storage capacity and energy, fresh data collected by devices often overwrites outdated data and establishes a heterogeneous data distribution. This causes the global model to forget outdated data's characteristics (i.e., catastrophic forgetting). Existing methods incorporate knowledge distillation into FL (i.e., federated distillation, FD) to extract and integrate characteristics from both fresh and outdated data, but they use fixed distillation temperatures for different devices, which overlooks that fixed distillation temperatures cannot match the heterogeneous data distribution on different devices and degrades global model accuracy. To this end, we propose a Federated Dynamic Decoupled Distillation method based on Logits distribution (Fed3DL). Specifically, Fed3DL utilizes decoupled distillation to mitigate catastrophic forgetting. To alleviate the impact of heterogeneous data distributions, Fed3DL novelly builds an adaptive temperature-aware mechanism to dynamically adjust the distillation temperature of each device based on the distribution of Logits. Additionally, Fed3DL introduces a regularization term into the local distillation loss to reduce inter-class characteristics disparity and improve model accuracy. Experiments on two datasets show that compared with the best of the 5 baselines, Fed3DL can improve the global model accuracy by an average of 3.40 %, reduce the forgetting rate by an average of 4.98 %, and achieve the lowest inter-class accuracy disparity.