Poster Abstract: FedMod: Towards Cross-modal Training for Heterogeneous Federated Learning Systems

Federated learning in multi-modal systems faces challenges due to modality heterogeneity, where edge devices have different sensor setups. Labeling multi-modal data is labor-intensive and impractical, leading to the label scarcity issue on edge clients. This paper presents a novel semi-supervised federated learning framework to address these issues. It uses complementary data, like RGB images and depth sensors, with a pseudo-labeling algorithm to improve cross-modal learning. Applied to the human action recognition task, the framework outperforms baselines. It enables efficient federated learning, handling labeling difficulties and missing modalities, offering robust performance in real-world scenarios.