Energy‐Efficient Optimization for RSMA‐Based Integrated Satellite–Terrestrial Networks Under Shadowed‐Rician Fading

ABSTRACT This paper develops a unified analytical and optimization framework for rate‐splitting multiple access (RSMA) in integrated satellite–terrestrial networks operating over shadowed‐Rician fading channels. Closed‐form integral expressions for the ergodic sum rate are derived using Meijer‐ functions, explicitly accounting for imperfect successive interference cancellation (SIC), multibeam satellite gains, and power splitting between common and private streams. Building on these expressions, a gradient‐projection algorithm combined with Dinkelbach's method is proposed to optimize the power‐split coefficients under quality‐of‐service and SIC constraints, jointly enhancing spectral and energy efficiency. Numerical and Monte Carlo results confirm the analytical accuracy and show that the optimized RSMA design achieves approximately 8%–20% ergodic sum‐rate and 10%–18% energy‐efficiency gains over non‐orthogonal multiple access (NOMA) and orthogonal multiple access (OMA) baselines across both average and heavy shadowing conditions. The proposed framework provides implementation‐oriented design insights into power‐split adaptation and SIC robustness for future 6G non‐terrestrial networks.