FedBRB: A Solution to the Small-to-Large Scenario in Device-Heterogeneity Federated Learning

Recently, the success of large models has demonstrated the importance of scaling up model sizes. However, it is difficult to directly train large models locally on multiple mobile devices due to their intrinsic computational constraints. To address this challenge, it becomes a crucial need to train larger global models by training small local models on devices. As a distributed learning approach, federated learning (FL) allows multiple devices to train models locally and aggregate them to form the global model by sharing the updated parameters with the server, thus enabling the co-training of models. This promising feature has spurred an increasing interest in exploring the collaborative training of large models. Despite the advent of existing device-heterogeneity FL approaches, they still have limitations in fully covering the parameter space of the global model. To fill this gap, we propose a novel approach called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FedBRB</b> (<underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</u>lock- wise <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</u>olling and weighted <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</u>roadcast). The core idea of FedBRB is to utilize local models of small devices to train all modules of a large global model and broadcast the trained parameters to the entire space, thereby enabling faster information sharing. This approach not only improves training efficiency but also fully utilizes limited computational resources. Experiments demonstrate that FedBRB can produce significant performance gains, achieving state-of-the-art results. Additionally, this paper provides theoretical and experimental analyses of FedBRB convergence, thereby paving a theoretical ground and providing practical guidance for further research and application of the FedBRB method.