Joint Layer-wise Caching and Request Routing for Serverless Inference Acceleration at the Edge

In recent years, the serverless paradigm has been introduced into edge computing, enhancing resource utilization. However, it has also introduced the issue of cold start, which hinders low-latency responses. This cold start problem is significantly exacerbated when it comes to Artificial Intelligence (AI) applications, due to the necessity of loading bulky deep learning (DL) model files. Existing methods typically address the cold start issue by caching complete containers, which is inefficient in resource-constrained edge servers. This is because massive DL models may not fit in the limited cache space or lead to fragmentation of the cache memory. To solve the above problem, we seek a layer-wise model caching method that regards the DL model as a chain structure of multiple layers and caches a part of the model layers. However, heterogeneity in cold starts and edge computing’s complex environments pose challenges in selecting model layers and servers. In this paper, we investigate the joint layer-wise caching and request routing problem to minimize application completion time (ACT). We propose an approximation algorithm with a provable performance guarantee. Finally, intensive trace-driven simulations show that our algorithm achieves 3.03× speedup in terms of ACT reduction.