COOL: A Cloud-Fog Federated Learning System With Multimodal Isolated Client Data in Edge Networks

Cloud-fog Federated Learning (FL) is promising for collaborative model training in large-scale edge networks. In cloud-fog FL with constrained communication resources, selecting high-quality client models for aggregation is critical to boost the global model. However, the clients in real world hold multimodal and heterogeneous data, while existing selection strategies rarely consider the imbalance of communication cost and model convergence rates across modalities, thus seriously degrading the efficiency of model aggregation. Moreover, most previous multimodal fusion methods require aligned multimodal samples. However, the data of modality-heterogeneous clients may be isolated and unaligned in FL, so these previous methods cannot be applied to such scenarios, thereby hindering the knowledge fusion across various modalities. To address the above issues, we propose a cloud-fog FL system named COOL, which achieves <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unimodal aggregation</i> at the fog layer and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multimodal fusion</i> at the cloud layer. First, we propose a Modality-aware Online Client Selection (MOCS) strategy to assist unimodal aggregation. Unlike previous selection strategies, MOCS realizes the dynamic selection budget allocation for various modalities by monitoring the convergence gap of modalities, thus striking the performance balance among modality-heterogeneous clients. Second, to overcome the limitation of previous methods that require aligned multimodal data, we propose a Multimodal Fusion strategy with Feature Synthesis (MFFS). MFFS realizes multimodal fusion with isolated samples via adaptive feature synthesis and cross-modal attention training, thus building a more powerful multimodal predictor while preserving the client data privacy. Finally, experimental results demonstrate that COOL has superior performance compared to the existing algorithms.