Autoencoder-Based MIMO Cooperative Communications With Quantize–Forward Relaying

This paper investigates an autoencoder-based quantize–forward (QF) relay system that includes a source, a destination, and a relay, each equipped with multiple antennas. The existing phase quantization (PQ) algorithm at the relay has limitations in capturing the amplitude differences of received signals, leading to performance saturation with increasing quantization bits. To address these limitations, we propose a novel relay algorithm, amplitude-phase quantization (APQ), which quantizes both the phase and the amplitude. Moreover, we introduce neural networks into the relay process, resulting in PQ with neural networks (PQNN) and APQ with neural networks (APQNN), which is expected to further improve system performance at the expense of additional computational load at the relay. We also propose a sub-message one-hot encoding method and a retraining approach for the worst-performing sub-message to reduce computational complexity and improve performance in autoencoder-based systems. Simulation results demonstrate that the autoencoder-based QF relay system, with various relay algorithms and the sub-message one-hot encoding method, achieves excellent performance with reduced memory usage at the relay and significantly reduced complexity at the source and destination.