Orchestrating Communication and Computation Efficiency in Edge-Based Knowledge Distillation

With the rapid development of edge computing, knowledge distillation is an effective method to overcome the challenges of training high-performance models with limited computational resources on the edge side. However, a naive implementation of knowledge distillation could cause resource contention on edge devices, specifically in the communication process of receiving distilled “dark knowledge” while simultaneously performing knowledge transfer, both competing for the limited resources. This paper delves into the system-level challenges of resource contention in edge-based knowledge distillation. In particular, we focus on the resource competition caused by the communication and computation processes. To address this issue, we first formulate the problem of edge-based knowledge distillation. Then we propose a method to orchestrate resources between computation and communication in an approximately optimal manner by employing a dynamic resource allocation strategy. Through simulation experiments, we demonstrate the effectiveness of the proposed scheme via evaluating against random and priority-based resource allocation strategies.