On Using BERT Embeddings for Text Semantic Communication

The current technological advancements carry stringent communication requirements. These requirements cannot be achieved by current networks. Therefore, a new paradigm called semantic communication is proposed. Semantic communication is the transmission of the meaning of data. Different semantic communication algorithms were proposed in literature, but they are complex and non-universal. In this work, a text semantic communication algorithm will be modified to be simple and universal. The modified algorithm uses a transformer network with 1 encoder layer and 3 decoder layers. The embedding layer uses pretrained embeddings from a Bidirectional Encoder Representations from Transformers (BERT) model. The algorithm is trained to optimize both the cross-entropy loss and the mutual information. Training and testing are done using the European parliament proceedings dataset. The performance of the modified algorithm is comparable to the original algorithm although it uses 50% of the layers. This work indicates the potential of using pretrained embeddings to simplify and enhance universality of existing semantic communication algorithms.