Collaborative Deep Reinforcement Learning in 6G Integrated Satellite-Terrestrial Networks: Paradigm, Solutions, and Trends

6G integrated satellite-terrestrial networks (ISTN) deploy a large number of satellites in outer space to improve network capacity and computing service coverage. However, the “first come, first served” rule for satellite frequency and orbit resources has led to severe scarcity of available spectrum and significant interference in large-scale satellite constellations. Moreover, the complex and constantly changing network topology, due to the high-speed motion of different orbital satellites, makes it challenging to provide efficient ubiquitous computing services. The traditional single type artificial intelligence (AI) method is insufficient for such complicated scenarios. Therefore, this article proposes a collaborative deep reinforcement learning (DRL) paradigm for ISTN to achieve intelligent multi-orbit spectrum and computing resource management, which is achieved through three progressively advanced solutions. First, the intelligent satellite scheduling can be implemented by DRL, which is combined with the graph convolutional network (GCN) to analyze the ISTN topology features. Then, multi-agent DRL is used for distributed modeling of satellites and ground users, and federated learning (FL) is integrated to achieve rapid resource allocation and interference coordination. Furthermore, meta-DRL is employed to accomplish adaptive computing offloading, avoiding the model retraining caused by ISTN changes. Finally, we use a typical use case to validate the effectiveness of the proposed solutions, and the open research trends with our proposed paradigm are discussed for the future ISTN evolution.