Layer Streaming Distillation

Distributed machine learning offers significant advantages over centralized approaches, including improved accessibility, memory efficiency, and performance. However, despite these benefits, distributed training remains underused, often sacrificed in favor of data security-centric methods such as federated learning. A major barrier to wider adoption is the lack of effective algorithms that efficiently leverage distributed compute. Current training methods often rely on bandwidth-heavy all-gather operations and global gradient dependencies, making distributed training challenging. In this paper, we propose a novel approach to distributing layer-wise knowledge distillation. By carefully routing data, passing layer parameters instead of activations, and eliminating cross-device gradient dependencies, our method significantly reduces training wall time compared to data-parallelization. Furthermore, our approach reduces communication bandwidth, improves memory utilization, and enables flexible resource allocation.