When do e-commerce prediction models fail over time? cross-task evidence on temporal reliability and model maintenance

Machine-learning models in e-commerce are often evaluated on randomly partitioned historical data, whereas deployment requires prediction on future observations under changing populations and information availability. We examine temporal reliability and model maintenance across four public dataset–task settings: late-delivery prediction among eventually delivered Olist orders, matched 24-h cart-to-purchase prediction in Retailrocket and Synerise RecSys 2025, and 30-day repeat-purchase prediction in Online Retail II. Sequential analyses restrict historical features, model updating, and recalibration to information and labels available at each prediction or deployment time. Random-to-future differences varied substantially: Olist showed pronounced deterioration, whereas the other settings showed smaller or qualitatively different changes. In the primary Olist XGBoost analysis, cumulative retraining improved ranking and modestly improved Brier accuracy, while sliding-window retraining improved ranking but worsened probability reliability. Cross-model, matched-exposure, and sensitivity analyses showed that maintenance effects depended on setting, period, model family, and design. Post-hoc recalibration did not provide a consistently reliable repair, and same-base calibration effects differed from net maintenance-policy effects. The matched comparisons do not identify causal task or duration effects. These findings support forward-in-time evaluation that assesses ranking, probability quality, label availability, and maintenance effects separately rather than inferring future reliability from offline performance or task identity.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-14
DOI
https://doi.org/10.1038/s41598-026-80365-9
Primary Topic
Customer churn and segmentation
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

When do e-commerce prediction models fail over time? cross-task evidence on temporal reliability and model maintenance

Yujiao Wang, Zongjun Lan
Scientific Reports
Customer churn and segmentation
article

When do e-commerce prediction models fail over time? cross-task evidence on temporal reliability and model maintenance

Yujiao Wang, Zongjun Lan
article en

Abstract

Machine-learning models in e-commerce are often evaluated on randomly partitioned historical data, whereas deployment requires prediction on future observations under changing populations and information availability. We examine temporal reliability and model maintenance across four public dataset–task settings: late-delivery prediction among eventually delivered Olist orders, matched 24-h cart-to-purchase prediction in Retailrocket and Synerise RecSys 2025, and 30-day repeat-purchase prediction in Online Retail II. Sequential analyses restrict historical features, model updating, and recalibration to information and labels available at each prediction or deployment time. Random-to-future differences varied substantially: Olist showed pronounced deterioration, whereas the other settings showed smaller or qualitatively different changes. In the primary Olist XGBoost analysis, cumulative retraining improved ranking and modestly improved Brier accuracy, while sliding-window retraining improved ranking but worsened probability reliability. Cross-model, matched-exposure, and sensitivity analyses showed that maintenance effects depended on setting, period, model family, and design. Post-hoc recalibration did not provide a consistently reliable repair, and same-base calibration effects differed from net maintenance-policy effects. The matched comparisons do not identify causal task or duration effects. These findings support forward-in-time evaluation that assesses ranking, probability quality, label availability, and maintenance effects separately rather than inferring future reliability from offline performance or task identity.

Scientific Reports
Zhejiang Guangsha Vocational and Technical University of Construction (CN), Guangzhou College of Technology and Business (CN)
Customer churn and segmentation
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
When do e-commerce prediction models fail over time? cross-task evidence on temporal reliability and model maintenance — Yujiao Wang, Zongjun Lan · Scientific Reports (2026) | TGRS Research Map | TGRS