Aggregate based light incremental sharding for efficient embedding table management for recommender systems

This work develops a novel method for efficient dynamic embedding table sharding during recommendation inference. Although existing works have developed various sharding methods to reduce network overhead, they often focus on training-centric approaches and overlook inference-specific challenges such as evolving co-occurrence patterns and latency sensitivity. As such, these methods can be suboptimal for real-time recommendation inference scenarios. In this paper, we propose a new method named ALIS, short for Aggregate based Light Incremental Sharding, to address these challenges. ALIS features two key mechanisms: aggregate based sharding, which enhances sharding stability and quality by integrating historical and current sharding results, and light incremental sharding, which reduces dynamic sharding costs by combining iterative and statistics-based approaches. The evaluation of ALIS demonstrates its superior performance over the state-of-the-art method in reducing network overhead and improving inference efficiency. Both quantitative results and qualitative analysis verify the superiority and rationality of our ALIS method.