On Sampling Reliability in Offline Recommender Evaluation

When online testing is impractical or risky, offline evaluation remains the primary mechanism for benchmarking recommender systems. Yet it is inherently affected by two sources of distortion: exposure bias, arising from system-driven item visibility, and sampling bias, introduced when evaluation relies on a subset of logged items rather than the full catalog. Prior work has proposed techniques to mitigate sampling bias. However, these techniques are typically assessed on static logs. Such assessments do not establish whether the techniques support reliable model comparisons under varying exposure conditions or against true user preferences. In this paper, we systematically study how logging and sampling choices jointly affect the reliability of offline evaluation. Using a fully observed dataset as ground truth, we simulate diverse exposure regimes and evaluate common sampling strategies along four dimensions: resolution (model separability), fidelity (agreement with full evaluation), robustness (stability under exposure bias), and predictive power (alignment with ground truth). We further examine whether these criteria align or reveal trade-offs in sampler performance. To strengthen external validity, we replicate key analyses on two additional benchmark datasets with fixed missing-completely-at-random (MCAR) test exposure. Our findings characterize when sampling distorts evaluation outcomes and provide controlled empirical evidence regarding how sampling strategies behave under different exposure and sparsity conditions.

Authors

Institutions

Publication Details

Journal
ACM Transactions on Recommender Systems
Published
2026-11-13
DOI
https://doi.org/10.1145/3862855
Primary Topic
Recommender Systems and Techniques
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

On Sampling Reliability in Offline Recommender Evaluation

Bruno Laporais Pereira, Rodrygo L. T. Santos, Alan Said
1 links
ACM Transactions on Recommender Systems
Recommender Systems and Techniques
article

On Sampling Reliability in Offline Recommender Evaluation

Bruno Laporais Pereira, Rodrygo L. T. Santos, Alan Said
article en

Abstract

When online testing is impractical or risky, offline evaluation remains the primary mechanism for benchmarking recommender systems. Yet it is inherently affected by two sources of distortion: exposure bias, arising from system-driven item visibility, and sampling bias, introduced when evaluation relies on a subset of logged items rather than the full catalog. Prior work has proposed techniques to mitigate sampling bias. However, these techniques are typically assessed on static logs. Such assessments do not establish whether the techniques support reliable model comparisons under varying exposure conditions or against true user preferences. In this paper, we systematically study how logging and sampling choices jointly affect the reliability of offline evaluation. Using a fully observed dataset as ground truth, we simulate diverse exposure regimes and evaluate common sampling strategies along four dimensions: resolution (model separability), fidelity (agreement with full evaluation), robustness (stability under exposure bias), and predictive power (alignment with ground truth). We further examine whether these criteria align or reveal trade-offs in sampler performance. To strengthen external validity, we replicate key analyses on two additional benchmark datasets with fixed missing-completely-at-random (MCAR) test exposure. Our findings characterize when sampling distorts evaluation outcomes and provide controlled empirical evidence regarding how sampling strategies behave under different exposure and sparsity conditions.

ACM Transactions on Recommender Systems
Universidade Federal de Minas Gerais (BR), University of Gothenburg (SE)
Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, Conselho Nacional de Desenvolvimento Científico e Tecnológico, Fundação de Amparo à Pesquisa do Estado de Minas Gerais
Recommender Systems and Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon