Producer-Side Experiments Based on Counterfactual Interleaving Designs for Online Recommender Systems

Recommender systems play a crucial role in online platforms, providing personalized recommendations for purchases, content consumption, and interpersonal connections. These systems involve two sides: producers (sellers, content creators, service providers, etc.) and consumers (buyers, viewers, customers, etc.). To optimize online recommender systems, A/B tests serve as the golden standard for comparing different ranking models and evaluating their impacts on both sides. While consumer-side experiments are relatively straightforward to design and commonly employed to assess ranking changes' effects on the behavior of consumers (buyers, viewers, etc.), designing producer-side experiments for an online recommender/ranking system is notably more complex. This complexity arises from the necessity of ranking producer items in the treatment and control groups by different models and then merging them into a unified ranking for presentation to each consumer. Existing design solutions in the literature lack rigorous guiding principles, leading to ad hoc approaches. In this paper, we address the limitations of current methods and propose the principles of consistency and monotonicity for designing producer-side experiments in online recommender systems. Building upon these principles, we also present a systematic solution based on counterfactual interleaving designs to accurately measure the impacts of ranking changes on the producers (sellers, content creators, etc.).