Sparse Autoencoder-Based Deep Learning Approach for Enhancing End-to-End Communication System Performance

Deep learning is increasingly adopted in future communication systems to meet requirements within constrained resources. End-to-end (E2E) autoencoder models leverage deep neural networks for effective communication but often suffer from performance degradation due to overfitting. To overcome this challenge, we propose the use of a sparse autoencoder (SAE). The SAE captures essential features by enforcing sparsity, enhancing model generalization. Building on this approach, we propose an SAE-based E2E communication system model designed for M-ary Phase Shift Keying (MPSK) and M-ary Quadrature Amplitude Modulation (MQAM) constellations over an Additive White Gaussian Noise (AWGN) channel. We train the SAE under varying signal-to-noise ratios (SNRs) and evaluate its Block Error Rate (BLER) versus testing SNR performance. Results show improved BLER at lower training SNRs, as the model better learns noise behavior. The proposed system outperforms a conventional BPSK system using maximum likelihood detection and a baseline E2E autoencoder, demonstrating the effectiveness of sparsity constraints. Additionally, we assess its performance considering model loss function, further validating its robustness.