Optimizing Knowledge Transfer Graph for Deep Collaborative Learning

Knowledge transfer among multiple networks, using predicted probabilities or intermediate-layer activations, has evolved significantly through extensive manual design, ranging from simple teacher–student approaches (for example, knowledge distillation) to bidirectional cohort methods (for example, deep mutual learning). However, key factors such as network size, the number of networks, transfer direction, and loss function design interact in complexways and limit conventional methods to exploring only a narrow range of possible combinations. In this paper, we propose a novel training method that enables more flexible and diverse combinations of knowledge transfer strategies. Specifically, we introduce a graph-based representation called the knowledge transfer graph, which provides a unified framework for representing a wide variety of knowledge transfer strategies. Furthermore, we propose specialized loss functions that incorporate five distinct gate functions to dynamically control gradient propagation and thereby enable a richer set of knowledge transfer strategies. By systematically searching the structure of the knowledge transfer graph, our method automatically discovers more effective knowledge transfer strategies than those designed manually. Experimental results on eleven datasets demonstrate that our approach significantly enhances performance and reliably identifies effective graph structures.