TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learning

In federated learning (FL), local personalization of models has received significant attention, yet personalized fine-tuning of foundation models remains a significant challenge. In particular, there is a lack of understanding in the literature on how to fine-tune and personalize foundation models in settings that are heterogeneous across clients not only in data, but also in tasks and modalities. To address this gap, we propose TAP (Two-Stage Adaptive Personalization), which has two key features: (i) leveraging mismatched model architectures between the clients and server to selectively conduct replacement operations when it benefits a client's local tasks; (ii) engaging in post-FL knowledge distillation for capturing beneficial general knowledge without compromising personalization. In developing TAP, we introduce the first convergence analysis of federated foundation model training at the server under its modality-task pair architecture, and demonstrate that as the number of modality-task pairs increases, its ability to cater to all tasks suffers. Through extensive experiments, we demonstrate the effectiveness of our proposed algorithm across a variety of datasets and tasks in comparison to state-of-the-art federated personalization baselines.