Intelligent hybrid detector for throughput enhancement in optical M-MIMO networks for 512-QAM and 1024-QAM

Abstract This paper proposes an Intelligent Minimum Mean Square Error–Deep Neural Network (IMMSE–DNN) hybrid signal detector for optical Multiple-Input Multiple-Output (MIMO) systems employing high-order modulation schemes. The aim is to ensure reliable detection for 64-Quadrature Amplitude Modulation (64-QAM), 256-QAM, 512-QAM, and 1024-QAM, where conventional detectors experience severe degradation due to noise enhancement, interstream interference, and nonlinear impairments. The proposed framework integrates a linear Minimum Mean Square Error (MMSE) front-end for effective interference suppression with a Deep Neural Network (DNN)-based nonlinear refinement stage to mitigate residual distortions. Simulation results obtained using MATLAB demonstrate that the proposed IMMSE–DNN consistently outperforms conventional MIMO, Zero-Forcing Equalizer (ZFE), MMSE, Maximum Likelihood Detection (MLD), autoencoder, and standalone DNN-based detectors. At a Bit Error Rate (BER) of 10 −3 , the proposed detector achieves SNR gains of approximately 4–5 dB for 64-QAM, 5–6 dB for 256-QAM, 6–7 dB for 512-QAM, and up to 8 dB for 1024-QAM compared to MMSE detection. Additionally, training accuracy analysis shows that the IMMSE–DNN converges rapidly, achieving nearly 90 % accuracy within 20 epochs, highlighting its robustness and suitability for next-generation high-capacity optical MIMO networks.