DL-AQUA: Deep-Learning-Based Automatic Quantization for MMSE MIMO Detection

Directly affecting both error performance and complexity, quantization is critical for MMSE MIMO detection. However, naively pruning quantization levels is not advised as it typically causes performance degradation. In practice, unified quantization (UQ) across different parameters based on empirical judgments or numerical simulations is mostly employed, as it circumvents sophisticated design and implementation efforts. Though UQ leaves room for performance/complexity improvements, manually fine-tuned quantization necessitates substantial efforts and is usually prohibitive for practical usage. To automate the quantization process, this letter first proposes a Deep-Learning based Automatic QUAntization (DL-AQUA) framework for MMSE detection. Multi-step optimization techniques, involving 1) gradient based search, 2) adaptive penalty based tuning, and 3) probability density function aware calibration, help to attain an agile yet fine-tuned quantization configuration. For <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8\times 128$ </tex-math></inline-formula> MIMO detection, DL-AQUA successfully achieves 35.06% quantization reduction than UQ without penalizing performance. Even benchmarking against the recent AHPQ technique, the relative gains are 14.53%. The robustness of DL-AQUA has been demonstrated across different antenna and SNR combinations. It should be noted that the immediate application of DL-AQUA is not limited to MMSE detection, but also extends to other quantization-sensitive DSP algorithms.