A Practitioner-Oriented Review of Reinforcement Learning for the Joint Optimization of Ads and Organic Content in Large-Scale Recommender Systems
Balancing advertisements with organic content is a challenge for large-scale recommendation platforms. Ads are central to platform revenue, but jointly serving them with organic content creates trade-offs between monetization and user experience. Many production systems frame these decisions as supervised learning problems that optimize immediate ad clickthrough, estimating immediate outcomes under a historical logging policy while treating decisions as independent. Formulating joint ad and organic ranking as long-horizon optimization over action sequences, under policy-induced state evolution, remains an open research problem. Reinforcement Learning (RL) offers a principled framework for modeling recommendation as sequential decision-making under uncertainty, but at scale it must represent high-dimensional state and action spaces, specify robust reward functions, explore safely, and manage substantial computational cost. This practitioner-oriented survey synthesizes prior RL work on the joint optimization of ads and organic content, distilling design choices, trade-offs, and evaluation practices through a unifying taxonomy that organizes each system by how it formulates the Markov Decision Process (MDP) in terms of state, action, and reward, and learns its policy. We find that industrial systems favor hybrid architectures and offline or constrained RL, such as contextual bandits and counterfactual policy evaluation, that learn from logged data while balancing scalability, safety, and interpretability.
Authors
- Yuri Lawryshyn (ORCID: https://orcid.org/0009-0005-8898-5821)
- Armando Ordorica (ORCID: https://orcid.org/0009-0009-8392-8691)
Institutions
- University of Toronto (CA)
Publication Details
- Journal
- ACM Transactions on Recommender Systems
- Published
- 2026-11-17
- DOI
- https://doi.org/10.1145/3859366
- Primary Topic
- Recommender Systems and Techniques
- Type
- article
- Field-Weighted Citation Impact
- 0.00