HRL-Based Proactive Caching Scheme for Vehicle–Edge–Cloud Collaborative System Applications

Proactive caching scheme proactively caches data before the request comes, which is essential for optimizing of the resource utilization and ensuring the stability of cached contents. However, current proactive caching methods suffer from high computational complexity and often fall into suboptimal solutions. This study proposes a hierarchical reinforcement learning (HRL)-based proactive caching scheme to determine whether to update the caching queue and how to update the caching queue. Specifically, we build a bi-level optimization model to maximize the content cache hit ratio while minimizing the content cache cost. We formulate the upper-level and lower-level optimization problems as Markov decision process (MDP) models and employ double deep Q-learning methods to solve the optimal caching strategy. In particular, the content history request features and the user preference generated by the collaborative filtering-based recommendation system are used as the state inputs of caching decisions. In addition, we design a two-stage curriculum learning paradigm for training the agents from easy tasks to hard tasks. At the first stage, a greedy decision tree with a posterior knowledge is designed to provide the upper-level decisions for training the lower-level agent in the easy task. Then, we train all agents together in the hard task while freezing the trained lower-level agent. We conduct both the simulation and on-road experiments to validate the effectiveness of our method, and results indicate that our method outperforms the baseline methods by 2.2%–37.2% in terms of cache hit ratio and 75.4%–77.6% in terms of cache cost.