EM2FL: An Embedding Multimodal Fusion Federated Learning Approach to Optimize Privacy and Performance

Multimodal federated learning enhances model performance by integrating data from different modalities. However, this integration also introduces new privacy leakage risk associated with cross-modal correlation. Existing differential privacy methods are primarily designed for single-modal data, making them ineffective in multimodal scenarios, while noise addition often compromises data utility. In this paper, we propose an Embedding Multimodal Fusion Federated Learning Approach, called EM2FL. This approach leverages horizontal multimodal federated learning to collaboratively train global models across multiple data platforms without exposing raw private data. Instead of applying differential privacy to each modality-specific encoder separately, we implement it on the mixed embedding module that combines multiple modalities. We tested our approach on 3 distinct models and validated its effectiveness for the combination of image and user-item pair modalities. Experimental results show that our method improves performance by 10-20% compared to traditional differential privacy methods while maintaining a similar level of privacy protection. This demonstrates that EM2FL effectively balances privacy and performance through selective noise addition.