A Hybrid Network Integrating MHSA and 1D CNN–Bi-LSTM for Interference Mitigation in Faster-than-Nyquist MIMO Optical Wireless Communications

To mitigate inter-symbol interference (ISI) caused by Faster-than-Nyquist (FTN) technology in a multiple input multiple output (MIMO) optical wireless communication (OWC) system, we propose an ISI cancellation algorithm that combines multi-head self-attention (MHSA), a one-dimensional convolutional neural network (1D CNN), and bi-directional long short-term memory (Bi-LSTM). This hybrid network extracts data features using 1D CNN and captures sequential information with Bi-LSTM, while incorporating MHSA to comprehensively reduce ISI. We analyze the impact of antenna numbers, acceleration factors, wavelength, and turbulence intensity on the system’s bit error rate (BER) performance. Additionally, we compare the waveform graphs and amplitude–frequency characteristics of FTN signals before and after processing, specifically comparing sampled values of four-pulse-amplitude modulation (4PAM) signals with those obtained after ISI cancellation. The simulation results demonstrate that within the Mazo limit for selecting acceleration factors, our proposal achieves a 7 dB improvement in BER compared to the conventional systems without deep learning (DL)-based ISI cancellation algorithms. Furthermore, compared to systems employing a point-by-point elimination adaptive pre-equalization algorithm, our proposal exhibits comparable BER performance to orthogonal transmission systems while reducing computational complexity by 31.15%.