Task-oriented efficient communication in federated learning via regularized sparse randomized networks and NLQ

Federated learning (FL) has emerged as a promising framework for distributed machine learning, allowing edge devices to collaboratively train models without sharing local data. However, communication efficiency remains a critical bottleneck. This paper proposes a novel framework that integrates regularized sparse randomized networks to address this challenge. The core idea is to introduce a global probability mask, updated iteratively, that guides the formation of sparse sub-networks on edge devices. Each device optimizes local sub-networks by sampling binary masks based on the global probability mask, effectively reducing model complexity while maintaining predictive performance. The proposed method incorporates a regularization term into the loss function to enhance sparsity and mitigate redundancy in the transmitted binary masks. This approach balances the trade-off between communication overhead on the natural language query (NLQ) based tasks and model learning performance, significantly reducing bitrate requirements in uplink communication. Experimental results on the MNIST dataset, under both IID and non-IID settings, demonstrate the effectiveness of the proposed framework. Notably, the proposed method achieves comparable test accuracy to the existing approaches, such as FedPM, while reducing communication bitrate by up to 80% in certain configurations. Moreover, the analysis of different regularization parameters reveals a clear trade-off between the sparsity and accuracy, where a lower regularization can enhance convergence and generalization, achieving higher accuracy in fewer communication rounds, while higher values prioritize communication efficiency.