Scored on Empty Histories: A Pre-registered Two-Pass Study of Offline Recommender Evaluation, in Which Sampled Metrics Crowned a Random Ranker and Random Splits Inflated NDCG up to Sixtyfold

Offline evaluation is how recommender systems are usually compared before anyone is exposed to them, and it rests on shortcuts: metrics computed against a small sample of negatives, random train and test splits, and naive scoring of logs that a biased system produced. This study pre-registered a measurement of how far those shortcuts move the answer, on a dataset where the answer is known. It ran in two passes. The first pass (2026-11-07) executed a frozen design on KuaiRec, whose small matrix is fully observed, with MovieLens-1M in a replication role. Its temporal and random-split arms were valid and showed large effects: random splits inflated exact NDCG@10 for all four non-random rankers on KuaiRec, by about 60 times for ItemKNN (0.0563 against 0.00094), and sampled metrics overstated exact NDCG@10 by one to two orders of magnitude on both temporal datasets. But verification of that pass proved a protocol-level defect in its fully observed arm: the frozen training pool contained zero events by any of the 1,411 scored users, so ItemKNN, EASE, and iALS were scored on empty histories, returned identical all-zero score matrices, and produced tie-break artifacts (exact NDCG@10 of 0.0634 for all three) instead of measurements. Six verdicts that rested on those rows are void. The second pass (2026-11-08) rebuilt the arm inside the fully observed matrix, put a mandatory coverage gate in front of every score, and re-ran the whole study under a second frozen pre-registration. The gate passed on every arm (corrected-arm user coverage 1.0000, against 0.0000 for the first pass's construction measured by the same code), and all five of the first pass's standing verdicts replicated, with headline statistics matching to a maximum absolute difference of 3.38e-07. On the corrected arm, exact NDCG@10 over 1,381 users orders iALS (0.01353) above EASE (0.01120) above Random (0.00654) above ItemKNN (0.00079) above Popularity (0.00000), while sampled metrics put the uniform random ranker first; the rank-mapping correction cut the sampled mean absolute error from 0.0943 to 0.0262 and restored the exact ordering. The off-policy estimators that appeared to beat the naive estimator on the void arm (mean Kendall tau difference 0.68) did not beat it on the real one (IPS minus 0.06, SNIPS minus 0.04, clipped IPS minus 0.06, every interval covering 0). Thirteen hypotheses were decided in the second pass: nine CONFIRMED, four REFUTED, none left uncomputed, and the replication gate passed. The claims are deliberately narrow: two datasets, five rankers, one machine, and an off-policy arm whose exposure is simulated. Within those bounds, the standard shortcuts move offline scores by orders of magnitude and can reverse model rankings, the standard correction repairs most of that specific damage, and none of it is visible unless the evaluation first proves that the users it scores have histories at all.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24328913
Primary Topic
Recommender Systems and Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Scored on Empty Histories: A Pre-registered Two-Pass Study of Offline Recommender Evaluation, in Which Sampled Metrics Crowned a Random Ranker and Random Splits Inflated NDCG up to Sixtyfold

Jesun Ahmad Ushno
1 links
Zenodo (CERN European Organization for Nuclear Research)
Recommender Systems and Techniques
preprint

Scored on Empty Histories: A Pre-registered Two-Pass Study of Offline Recommender Evaluation, in Which Sampled Metrics Crowned a Random Ranker and Random Splits Inflated NDCG up to Sixtyfold

Jesun Ahmad Ushno
preprint en

Abstract

Offline evaluation is how recommender systems are usually compared before anyone is exposed to them, and it rests on shortcuts: metrics computed against a small sample of negatives, random train and test splits, and naive scoring of logs that a biased system produced. This study pre-registered a measurement of how far those shortcuts move the answer, on a dataset where the answer is known. It ran in two passes. The first pass (2026-11-07) executed a frozen design on KuaiRec, whose small matrix is fully observed, with MovieLens-1M in a replication role. Its temporal and random-split arms were valid and showed large effects: random splits inflated exact NDCG@10 for all four non-random rankers on KuaiRec, by about 60 times for ItemKNN (0.0563 against 0.00094), and sampled metrics overstated exact NDCG@10 by one to two orders of magnitude on both temporal datasets. But verification of that pass proved a protocol-level defect in its fully observed arm: the frozen training pool contained zero events by any of the 1,411 scored users, so ItemKNN, EASE, and iALS were scored on empty histories, returned identical all-zero score matrices, and produced tie-break artifacts (exact NDCG@10 of 0.0634 for all three) instead of measurements. Six verdicts that rested on those rows are void. The second pass (2026-11-08) rebuilt the arm inside the fully observed matrix, put a mandatory coverage gate in front of every score, and re-ran the whole study under a second frozen pre-registration. The gate passed on every arm (corrected-arm user coverage 1.0000, against 0.0000 for the first pass's construction measured by the same code), and all five of the first pass's standing verdicts replicated, with headline statistics matching to a maximum absolute difference of 3.38e-07. On the corrected arm, exact NDCG@10 over 1,381 users orders iALS (0.01353) above EASE (0.01120) above Random (0.00654) above ItemKNN (0.00079) above Popularity (0.00000), while sampled metrics put the uniform random ranker first; the rank-mapping correction cut the sampled mean absolute error from 0.0943 to 0.0262 and restored the exact ordering. The off-policy estimators that appeared to beat the naive estimator on the void arm (mean Kendall tau difference 0.68) did not beat it on the real one (IPS minus 0.06, SNIPS minus 0.04, clipped IPS minus 0.06, every interval covering 0). Thirteen hypotheses were decided in the second pass: nine CONFIRMED, four REFUTED, none left uncomputed, and the replication gate passed. The claims are deliberately narrow: two datasets, five rankers, one machine, and an off-policy arm whose exposure is simulated. Within those bounds, the standard shortcuts move offline scores by orders of magnitude and can reverse model rankings, the standard correction repairs most of that specific damage, and none of it is visible unless the evaluation first proves that the users it scores have histories at all.

Zenodo (CERN European Organization for Nuclear Research)
Recommender Systems and Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon