Effective Utilization of Large-scale Unobserved Data in Recommendation Systems

Ranking models play an important role in industrial recommendation systems. However, most ranking models are trained only with the observed items but used to retrieve all items in the entire space, which may suffer from the sample selection bias and the exposure bias. Inspired by the entire space learning framework, we carry out detailed data analyses on large-scale unobserved items and find that they contain quite a few "potentially-positive" samples. In this paper, we propose an "Extract and Transfer" (EAT) framework, utilizing quantities of unobserved items and other domains' data to construct more training data for ranking models. Specifically, we first extract "potentially-positive" samples and negative ones according to their ranking scores from the unobserved data, and then design an Entire Space Transfer Learning (ESTL) model to transfer knowledge between observed and unobserved samples, instead of directly mixing them together to avoid negative transfer. Experiments on production data collected from Taobao validate the proposed method's superiority. Besides, we have deployed EAT on the Taobao recommendation system, obtaining 6.22% IPV (Item Page View) and 3.77% CTR improvement. The code is available at https://github.com/Recommender1/EAT.git1.