Multi-Agent Federated $Q$-Learning Algorithms for Wireless Edge Caching

Edge caching is an increasingly vital technique in wireless networks, particularly needed to address users' repeated demands, including real-time traffic data and map access in vehicular communications. This paper presents a three-tier system for edge caching, integrating massive multiple-input multiple-output (mMIMO) networks. We consider a scenario with an extensive file library whose size is larger than the aggregated caching capacity of the small base stations (SBSs). Each SBS proactively caches files based on dynamic file popularity that is unknown in advance. A distinguishing aspect of our model is its file-wise approach to the cache instead of the conventional vector-wise methods based on aggregate popularity. This approach introduces additional computational challenges concerning the extensive file library. We formulated the optimization problem to maximize the long-term discounted cache hit rate while minimizing the delivery delay. For the solution, two multi-agent federated <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula>-learning algorithms are proposed. The first algorithm employs selective updates of the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula>-values of popular files to minimize computational overhead. The second algorithm incorporates linear function approximation (LFA) and tensor completion (TC) to streamline the updating process further, reducing the required parameter number. Through the real-world MovieLens dataset and compared with various baseline algorithms, simulations demonstrate that our proposed algorithm can reduce the delay by around 2.60--21.29%, improve the cache hit rate by around 5.71--66.42%, and reduce the computational complexity by at most 97.91%.