Dynamic Client Selection and Group-Balanced Personalization for Data-Imbalanced Federated Speech Recognition

Federated learning has been widely applied in automatic speech recognition. However, variations in speaker behaviors result in a significant data imbalance across client devices. Conventional federated speech recognition algorithms typically use fixed probabilities to select clients for each round in model training, often overlooking the disparities in data volume among clients. In reality, the substantial differences in data quantity can extend the training duration and compromise the stability of the global model. Moreover, models trained through federated learning on global data often fail to achieve optimal performance for individual local clients. While personalized federated learning strategies hold promise for enhancing model performance, the inherent diversity of speech data makes it challenging to apply state-of-the-art personalized methods effectively to speech recognition tasks. In this paper, a dynamic client selection algorithm is proposed to solve the problem of data disparities among different clients. It can be effectively combined with most federated learning algorithms and dynamically adjusts the selection probabilities of clients based on their dataset size during training. Experimental results demonstrate that this algorithm saved training time by 26% compared to traditional methods on public datasets while maintaining the equivalent model performance. To optimize the personalized federated learning, this paper proposes a novel group-balanced personalization strategy that fine-tunes groupings of clients based on their dataset size. The experimental results show that this algorithm brought a relatively 12% reduction in character error rate, while it did not increase computational costs. In particular, the group-balanced personalization effectively improved the model performance for clients with smaller datasets than local fine-tuning. The combination of dynamic client selection and group-balanced personalization significantly enhanced training efficiency and model performance.