Personalized Multi-Modal Federated Learning Over Heterogeneous Edge Devices

Multi-modal learning improves model robustness and accuracy by integrating complementary information from multiple modalities, addressing the limitations of uni-modal approaches in handling complex tasks. To address privacy and communication constraints, federated learning (FL) has been adopted for distributed multi-modal model training, where only model parameters instead of raw data are uploaded. Traditional multi-modal FL methods face performance degradation due to the intricate interplay of three types of modality heterogeneity: modality quantity, modality feature emphasis, and modality statistical distribution. In this paper, we propose HeteroPMMFL, a novel personalized multi-modal FL framework. HeteroPMMFL proposes a model decoupling approach to effectively address modality combination differences and designs personalized collaboration graphs to strengthen cooperation among similar clients. This framework is applicable to any multi-modal dataset and can be easily extended to accommodate various modality combinations. Experimental results show that HeteroPMMFL outperforms both traditional FedAvg and the state-of-the-art Harmony framework in model accuracy, under the presence of three types of modality heterogeneity.