An Integrated OTFS-NOMA Framework for Multi-Beam LEO Systems: Reliability and Capacity Analysis

Multi-beam low earth orbit (LEO) satellite communications, as an essential component for 6G systems, may encounter challenges from severe Doppler shifts and co-channel interference. This paper addresses a realistic problem in 6G-LEO systems, that is, how to meet the high-reliability demands of massive high-mobility terminals. We propose an integrated framework to exploit the synergy of non-orthogonal multiple access (NOMA) and orthogonal time frequency space (OTFS). OTFS modulation is employed to achieve full time-frequency diversity to combat Doppler shifts, while NOMA is used to accommodate more access requests. Specifically, within each beam, power domain superposition is applied to the delay-Doppler domain, enabling multiple terminals to share delay-Doppler grid resources. We analyze the performance of reliability, outage probability and ergodic capacity. Notably, we derive a novel closed-form expression to characterize the distribution of multi-beam interference with varying beam gains. Theoretical analysis and simulation results confirm that the proposed framework achieves a substantially lower outage probability compared to conventional OFDM schemes, with a system capacity improvement exceeding 11.9%.