Enhancing Collaborative Machine Learning in Resource-Limited Networks Through Knowledge Distillation and Over-the-Air Computation

Conventional collaborative machine learning (CML) faces significant challenges in resource-constrained environments, such as emergency scenarios with limited power, bandwidth, and computing resources, leading to increased communication delays and energy consumption. To address these issues, this paper introduces <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Air-CoKD</monospace>, a novel CML framework designed to reduce resource consumption and training latency while preserving model performance. <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Air-CoKD</monospace> leverages knowledge distillation (KD) to minimize data transmission by avoiding the direct sharing of model parameters. It also integrates over-the-air computation (AirComp) to aggregate local logits, optimizing bandwidth utilization. To address the dimensional differences in local logits caused by the unbalanced device data class, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Air-CoKD</monospace> employs orthogonal frequency division multiplexing (OFDM) to transmitting local logits for different target classes. To handle aggregation errors introduced by AirComp, we conduct a detailed analysis of error bounds. Specifically, we convert the Kullback-Leibler (KL) divergence, used in KD loss function, into a quadratic upper bound for precise error quantification and effective optimization. Based on these insights, we propose a strategy to manage bandwidth constraints, transmission power limits, and device energy budgets within <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Air-CoKD</monospace>. Extensive simulations demonstrate that <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Air-CoKD</monospace> surpasses state-of-the-art methods, effectively balancing training efficiency and model performance. The framework proves to be a robust solution for CML in resource-constrained networks.