Tensor Dynamic Fusion-Based Modality-Imbalanced Multimodal Federated Learning in Mobile Edge Computing for Consumer Applications

Mobile edge computing extends the capabilities of computation and storage from the cloud to the edge, enabling local data storage and processing. This computing paradigm complements federated learning (FL), as it satisfies the need for privacy protection while providing efficiency and quick response. In consumer applications, multimodal data is experiencing explosive growth. Therefore, multimodal FL has emerged as a crucial research area. However, applying traditional FL methods directly to multimodal data poses certain challenges. Additionally, issues like sensor failures result in modality-imbalanced data among clients, i.e., the number and type of data modalities between client devices vary. Hence, it is imperative to design specialized FL frameworks and models specifically tailored for analyzing multimodal data. To address these challenges, we propose a tensor fusion-based multimodal federated learning (TenMFL). First, a tensor dynamic fusion network is designed for addressing the challenge of multimodal data fusion. This approach separates the parameter forward computation and the modal fusion computation, effectively resolving issues related to client-side modality imbalance. Second, we introduce a hybrid fusion method that combines feature fusion and decision fusion to further exploit the complementarity of multimodal data. We evaluate TenMFL on three multimodal datasets, experimental results demonstrate our superior performances.