Sharded Machine Forgetting for User Data Deletion in the Ad Ranking Model Training Pipeline

Once a user requests that their data be deleted, their behavior data should not affect the training process anymore, but behavior data prior to their deletion should still have an impact on ranking models already deployed, and frequently retraining ranking models due to deletion requests is too expensive. In this work, we add SISA sharded forgetting training method to the training pipeline of the advertising ranking model. The training data is partitioned into 32 disjoins sets (shards) with the user hashes, each split into 8 segments of time, each checkpointed between segments. A DCN-V2 submodel is trained separately for each shard, and during inference, weighted average of the outcomes from submodels will be conducted. A deletion request only affects the shards that are deleted to be retrained from the time segment corresponding to the deletion request. The data employed included 40.43 million clicks by the testing experiments carried out on Avazu click logs and 25 million ratings by 162, 000 users provided by MovieLens 25M. The deletion requests were simulated to be requested from users at the rate of deletion of 0.05% users per day and forgetting effect evaluated using Likelihood Ratio Membership Inference Attack (LiRA). For the Avazu data set, the AUC went down from 0.7853 to 0.7821 (0.6% difference in log loss), training cost for one batch of deletion decreased from 6.4 GPU hours for a complete retraining to 0.31 GPU hours (95.2% decrease in training effort) and time from request to updated model was decreased from 7 days to 3.2 hours. Following the forgetting LiRA obtained a true positive rate of 0.11% at a 0.1% false positive rate for deleted users, which was not significantly different from control users who had not been involved in training (0.10%), and the gradient-ascending approximate forgetting method was still 1.9%. Two one-sided tests (TOST) showed that for non-deleted users, the difference in the prediction scores for the SISA model and the fully retrained model was not different by more than ±0.002 (p < 0.01). The larger the number of partitions, the lower the retraining cost (roughly proportional to 1/npartitions) while the AUC loss grew from 0.0011 to 0.0068, with 32 partitions being considered as a good compromise between cost and accuracy.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-20
DOI
https://doi.org/10.21203/rs.3.rs-11785586/v1
Primary Topic
Privacy-Preserving Technologies in Data
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Sharded Machine Forgetting for User Data Deletion in the Ad Ranking Model Training Pipeline

Que Yang, Yuang Hu, Yukun Du
2 links
Research Square
Privacy-Preserving Technologies in Data
preprint

Sharded Machine Forgetting for User Data Deletion in the Ad Ranking Model Training Pipeline

Que Yang, Yuang Hu, Yukun Du
preprint en

Abstract

Once a user requests that their data be deleted, their behavior data should not affect the training process anymore, but behavior data prior to their deletion should still have an impact on ranking models already deployed, and frequently retraining ranking models due to deletion requests is too expensive. In this work, we add SISA sharded forgetting training method to the training pipeline of the advertising ranking model. The training data is partitioned into 32 disjoins sets (shards) with the user hashes, each split into 8 segments of time, each checkpointed between segments. A DCN-V2 submodel is trained separately for each shard, and during inference, weighted average of the outcomes from submodels will be conducted. A deletion request only affects the shards that are deleted to be retrained from the time segment corresponding to the deletion request. The data employed included 40.43 million clicks by the testing experiments carried out on Avazu click logs and 25 million ratings by 162, 000 users provided by MovieLens 25M. The deletion requests were simulated to be requested from users at the rate of deletion of 0.05% users per day and forgetting effect evaluated using Likelihood Ratio Membership Inference Attack (LiRA). For the Avazu data set, the AUC went down from 0.7853 to 0.7821 (0.6% difference in log loss), training cost for one batch of deletion decreased from 6.4 GPU hours for a complete retraining to 0.31 GPU hours (95.2% decrease in training effort) and time from request to updated model was decreased from 7 days to 3.2 hours. Following the forgetting LiRA obtained a true positive rate of 0.11% at a 0.1% false positive rate for deleted users, which was not significantly different from control users who had not been involved in training (0.10%), and the gradient-ascending approximate forgetting method was still 1.9%. Two one-sided tests (TOST) showed that for non-deleted users, the difference in the prediction scores for the SISA model and the fully retrained model was not different by more than ±0.002 (p < 0.01). The larger the number of partitions, the lower the retraining cost (roughly proportional to 1/npartitions) while the AUC loss grew from 0.0011 to 0.0068, with 32 partitions being considered as a good compromise between cost and accuracy.

Research Square
University of Chicago (US), Carnegie Mellon University (US)
Privacy-Preserving Technologies in Data
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon