Multimodal Federated Learning Via Local-Global Fusion

The number of Internet of Things (IoT) devices across diverse domains in modern life has made them significant sources for collecting and analyzing multi-modal data. However, concerns about ownership and data privacy associated with IoT devices make data sharing among multiple devices impractical. Recently, multimodal federated learning has emerged as an innovative solution where each device client can collectively train a satisfactory local model without exchanging local data. Nevertheless, most existing multimodal federated learning approaches prioritize training a powerful global server model while neglecting the performance of local client models. In this context, this paper introduces FedAF, a multimodal Federated learning approach via feature Fusion with adversarial representation learning, aimed at enhancing local representations and thereby improving local client models. Specifically, using the trained global model, FedAF integrates the global representation of each local client's data into its local feature obtained from the corresponding client model. Furthermore, domain adversarial learning is employed to align global and local representations by minimizing the discrepancy between local and global encoders, compelling the global encoder to adapt to local tasks. Comprehensive experiments on two unimodal classifications and one multimodal retrieval dataset demonstrate that FedAF achieves state-of-the-art performance compared to other federated learning methods and significantly improves local client models while maintaining the satisfactory performance of the global server model.