SM-RS 2.0: User-perceived Qualities of Single- and Multi-Objective Recommender Systems

Recommender systems (RS) rely on interaction data between users and items to generate effective results. Originally, RS aimed solely at predicting items’ relevance, but additional (beyond-relevance) quality criteria gained increased attention over time. Objectives such as diversity, novelty, fairness, or serendipity are nowadays at the center of RS research and also among the core components in production systems. Naturally, to properly steer toward such objectives, the system has to gain an understanding of how the users perceive these objectives, to what extent they require them in the recommendations, and how they evaluate the sufficiency of the results w.r.t. these objectives. However, so far, there is no publicly available dataset that would capture all the necessary knowledge. This results in a half-blind algorithmic design and evaluation, where the importance of individual objectives or the metrics for their evaluation cannot be validated from the users’ perspective. To address this issue, we present SM-RS 2.0, an expansion of the original single- and multi-objective recommendations dataset. The dataset links the self-declared propensity toward individual objectives with impressions, item selections, and explicit evaluation of individual quality criteria. Together with the dataset, we also distribute an evaluation framework containing six rather unique tasks that are rarely available to conduct on existing RS datasets. These include impression-aware click prediction, predicting propensity toward individual objectives, construction of proportional recommendations, and predicting the user-perceived fulfillment of individual objectives as well as their overall satisfaction. The dataset is available at https://osf.io/wsakx .