Distributed Fine- Tuning of Foundation Models Over Heterogeneous Edge Devices

The synergy between Federated Learning (FL) and Foundation Models (FMs) holds great promise in enhancing privacy protection and improving the generalization capabilities of AI systems. However, the high computational and communication overhead of FMs hinders effective deployment in real-world scenarios. Although some pioneering research has proposed using proxy sub-Foundation Models (sub-FMs) to reduce the computational and communication costs when fine-tuning FMs in FL environments, it overlooks the challenges posed by heterogeneous mobile devices with varying computational and communication capabilities, and by dynamic changes in their operational conditions, which cause very long FL training delay. Motivated by these challenges, we propose a novel federated fine-tuning of Foundation Models design via adaptive pruning (FedFTAP). FedFTAP introduces a pruning method specifically designed for FMs, combined with parameter-efficient fine-tuning modules to enhance communication and computational efficiency. FedFTAP further addresses system heterogeneity and system dynamic changes by adaptively tailoring heterogeneous sub-FMs suitable for local training on mobile devices. Moreover, FedFTAP introduces a method for aligning heterogeneous sub-FMs with the global FM. The experimental results show that FedFTAP effectively reduces computational and communication costs in federated fine-tuning scenarios.