Sensitivity-aware Federated ASR Fine-Tuning with Balancing Accuracy and Network Cost

Federated learning has recently attracted attention as a method for training large language models (LLMs) across distributed edge devices. However, fine-tuning large Automatic Speech Recognition (ASR) models in federated environments, especially consisting of resource heterogeneity, remains a challenging task. In this paper, we propose a resource-efficient sensitivityaware federated fine-tuning approach for ASR tasks, SaFed, that implements automatic mixed-precision (AMP) quantization updates based on the Hessian sensitivity analysis of trainable model layers. We propose an explicit memory budget term in the training objective to reduce the resource usage of clients during the fine-tuning process. We rank parameter groups by their sensitivity to precision allocation, allowing us to perform AMP quantization. To preserve memory usage, we also add a penalty if the total memory footprint exceeds a specified resource budget. SaFed helps optimize the computational cost and memory footprint of edge FL training on resource-constrained client devices by effectively utilizing gradients, even with limited precision and computational capabilities. Experiments conducted on more than 400 simulated FL edge clients demonstrate noticeable improvements in model convergence and accuracy over traditional FL methods. We show that our proposed method helps to preserve accuracy on two different ASR fine-tuning tasks while reducing memory usage by $44 \%$.