Signal detection in optical NOMA waveforms for VLC-based satellite systems using higher-order modulation

Abstract Visible-light communication (VLC) and free-space optical wireless communication offer high-frequency platforms for high-rate satellite links, while nonorthogonal multiple access (NOMA) enables multiple users to share the same optical resource. This paper investigates signal detection in a two-user optical NOMA waveform for VLC-based satellite links using higher-order pulse-amplitude modulation (PAM). The proposed framework integrates power-domain superposition coding, free-space optical channel modeling with path loss and log-normal turbulence, direct photodetection, and successive interference cancellation (SIC). Four-, 16-, 32-, and 64-PAM schemes are evaluated in terms of bit-error rate (BER), spectral efficiency, power allocation, link distance, turbulence sensitivity, detector performance, and outage probability. At approximately 20 dB SNR, the BER increases from about 10 −4 for 4-PAM to 5 × 10 −3 for 16-PAM and 4 × 10 −2 for 64-PAM. Increasing the weak-user power coefficient from 0.55 to 0.70 reduces BER by nearly 87 %. At 14 dB SNR, MMSE-assisted SIC and ML detection reduce BER by approximately 60 % and 85 %, respectively, compared with conventional SIC. The results demonstrate the trade-off among modulation order, detection complexity, power allocation, and link reliability, supporting future optical satellite communication systems.