Scored on Empty Histories: A Pre-registered Two-Pass Study of Offline Recommender Evaluation, in Which Sampled Metrics Crowned a Random Ranker and Random Splits Inflated NDCG up to Sixtyfold
Offline evaluation is how recommender systems are usually compared before anyone is exposed to them, and it rests on shortcuts: metrics computed against a small sample of negatives, random train and test splits, and naive scoring of logs that a biased system produced. This study pre-registered a measurement of how far those shortcuts move the answer, on a dataset where the answer is known. It ran in two passes. The first pass (2026-11-07) executed a frozen design on KuaiRec, whose small matrix is fully observed, with MovieLens-1M in a replication role. Its temporal and random-split arms were valid and showed large effects: random splits inflated exact NDCG@10 for all four non-random rankers on KuaiRec, by about 60 times for ItemKNN (0.0563 against 0.00094), and sampled metrics overstated exact NDCG@10 by one to two orders of magnitude on both temporal datasets. But verification of that pass proved a protocol-level defect in its fully observed arm: the frozen training pool contained zero events by any of the 1,411 scored users, so ItemKNN, EASE, and iALS were scored on empty histories, returned identical all-zero score matrices, and produced tie-break artifacts (exact NDCG@10 of 0.0634 for all three) instead of measurements. Six verdicts that rested on those rows are void. The second pass (2026-11-08) rebuilt the arm inside the fully observed matrix, put a mandatory coverage gate in front of every score, and re-ran the whole study under a second frozen pre-registration. The gate passed on every arm (corrected-arm user coverage 1.0000, against 0.0000 for the first pass's construction measured by the same code), and all five of the first pass's standing verdicts replicated, with headline statistics matching to a maximum absolute difference of 3.38e-07. On the corrected arm, exact NDCG@10 over 1,381 users orders iALS (0.01353) above EASE (0.01120) above Random (0.00654) above ItemKNN (0.00079) above Popularity (0.00000), while sampled metrics put the uniform random ranker first; the rank-mapping correction cut the sampled mean absolute error from 0.0943 to 0.0262 and restored the exact ordering. The off-policy estimators that appeared to beat the naive estimator on the void arm (mean Kendall tau difference 0.68) did not beat it on the real one (IPS minus 0.06, SNIPS minus 0.04, clipped IPS minus 0.06, every interval covering 0). Thirteen hypotheses were decided in the second pass: nine CONFIRMED, four REFUTED, none left uncomputed, and the replication gate passed. The claims are deliberately narrow: two datasets, five rankers, one machine, and an off-policy arm whose exposure is simulated. Within those bounds, the standard shortcuts move offline scores by orders of magnitude and can reverse model rankings, the standard correction repairs most of that specific damage, and none of it is visible unless the evaluation first proves that the users it scores have histories at all.
Authors
- Jesun Ahmad Ushno (ORCID: https://orcid.org/0000-0002-6070-9025)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24328913
- Primary Topic
- Recommender Systems and Techniques
- Type
- preprint