Keep The Positive in The Loop: A Field Experimental Study of The Impact of Integrating Solicited Consumer Feedback in Content Recommendation Systems

Recommender systems mostly rely on observed user engagement behavior for training and optimization. Yet, behavioral feedback has known biases in capturing consumers' preferences, which necessitate the inclusion of actively solicited user feedback in recommendation systems. We report findings from a field experimental study of the impact of integrating solicited consumer feedback in content recommendation systems. In collaboration with a significant social media platform, a pop-up survey module was introduced to actively collect content consumers' feedback (satisfied/dissatisfied/uncertain). The solicited consumer feedback was used to adjust video recommendations following either the "boosting" or "filtering" strategy. The results show that the two methods for incorporating solicited consumer feedback have distinctive impacts. The boosting strategy increased content diversity and user activity, especially for new users, while the filter strategy decreased content diversity and user engagement. This research reveals the nuances in the design of the integration of "human" and "algorithm" in "keep-human-in-the-loop" recommender systems.