A Supernet-Only Framework for Federated Learning in Computationally Heterogeneous Scenarios

Federated learning is effective for Internet of Things data privacy and non-independent and identically distributed issues but not device heterogeneity. Neural Architecture Search can alleviate this by constructing multiple model structures to optimize federated learning performance across diverse edge devices. However, existing methods, whether lightweight networks or client grouping, face a tradeoff between scaling to larger federations and utilizing more powerful structures. We decompose residual network blocks, reformulating them as a Neural Architecture Search task. Furthermore, we propose a method for reinterpreting any sequential architecture into a supernet and developed a training pipeline tailored to this reinterpretated architecture, mitigating this frustrating tradeoff. We conduct pretraining on ImageNet1K and federated training on the CIFAR-100, CIFAR-10, and CINIC-10 datasets under both the ring-based federated learning and FedAvg framework. In less constrained environments, our method maintains performance comparable to another top-two method, which varies across experimental settings, while maintaining a margin of at least 1% Top-1 accuracy over the third-best method. Under balanced settings, our method outperforms the second-best approach by more than 1%, and this advantage increases to over 5% as the task difficulty further rises. Under the most challenging setting, our method outperformed AdaptiveFL, a state-of-the-art dynamic network method for federated learning, by 18.3% on CIFAR-100 with 100 clients under a ResNet backbone.