Synesthesia of Machines-Enabled Multi-Task Semantic Communication System

In recent years, there has been significant progress in semantic communication systems empowered by deep learning. It has greatly improved the efficiency of information transmission. Nevertheless, traditional semantic communication models still face challenges, particularly due to their single-task and single-modal orientation. Many of these models are designed for specific tasks, which results in limitations when applied to multi-task communication systems. Moreover, these models often overlook the correlations among different modal data in multi-modal tasks. It leads to an incomplete understanding of complex information, causing increased communication payload and diminished performance. To address these limitations, Synesthesia of Machines (SoM) provides an effective framework for fusing multi-modal data, capturing their complementary relationships. Inspired by SoM, we propose a SoM-enabled multi-task semantic communication (SoMMSC) framework. In contrast to traditional semantic communication approaches, SoMMSC can effectively handle various tasks across multiple modalities. Furthermore, we design a fusion module based on Bidirectional Encoder Representations from Transformers (BERT) for multi-modal fusion. By leveraging the powerful semantic understanding capabilities and self-attention mechanism of BERT, we achieve effective fusion of different modalities. We compare our model with multiple benchmarks. Simulation results show that SoMMSC outperforms these models in terms of both performance and communication payload.