Semantic Segmentation-Based Low-Rate Image Communication With Diffusion Models

Semantic communication aims to transmit the underlying semantic information of a signal from the sender to the receiver, where the key requirement is to ensure that the receiver reconstructs a signal semantically (almost) equivalent to the source. Conventional image communication methods lack effective mechanisms for preserving both semantic coherence and reconstruction quality, especially for low-rate scenarios. In this paper, we propose an alternative method, namely Instance-Consistent Image Communication (ICIC), for semantic segmentation-based low-rate image communication. Our method leverages segmented instances and detected captions as intermediate vehicles to transmit semantic information from the sender to the receiver. Once transmitted, the receiver is capable of utilizing the vehicles to regenerate images with off-the-shelf diffusion models. Compared to state-of-the-art methods, experimental results on the COCO dataset illustrate that our method obtains higher semantic accuracy and higher image quality at extremely low rates.