Sharded Machine Forgetting for User Data Deletion in the Ad Ranking Model Training Pipeline
Once a user requests that their data be deleted, their behavior data should not affect the training process anymore, but behavior data prior to their deletion should still have an impact on ranking models already deployed, and frequently retraining ranking models due to deletion requests is too expensive. In this work, we add SISA sharded forgetting training method to the training pipeline of the advertising ranking model. The training data is partitioned into 32 disjoins sets (shards) with the user hashes, each split into 8 segments of time, each checkpointed between segments. A DCN-V2 submodel is trained separately for each shard, and during inference, weighted average of the outcomes from submodels will be conducted. A deletion request only affects the shards that are deleted to be retrained from the time segment corresponding to the deletion request. The data employed included 40.43 million clicks by the testing experiments carried out on Avazu click logs and 25 million ratings by 162, 000 users provided by MovieLens 25M. The deletion requests were simulated to be requested from users at the rate of deletion of 0.05% users per day and forgetting effect evaluated using Likelihood Ratio Membership Inference Attack (LiRA). For the Avazu data set, the AUC went down from 0.7853 to 0.7821 (0.6% difference in log loss), training cost for one batch of deletion decreased from 6.4 GPU hours for a complete retraining to 0.31 GPU hours (95.2% decrease in training effort) and time from request to updated model was decreased from 7 days to 3.2 hours. Following the forgetting LiRA obtained a true positive rate of 0.11% at a 0.1% false positive rate for deleted users, which was not significantly different from control users who had not been involved in training (0.10%), and the gradient-ascending approximate forgetting method was still 1.9%. Two one-sided tests (TOST) showed that for non-deleted users, the difference in the prediction scores for the SISA model and the fully retrained model was not different by more than ±0.002 (p < 0.01). The larger the number of partitions, the lower the retraining cost (roughly proportional to 1/npartitions) while the AUC loss grew from 0.0011 to 0.0068, with 32 partitions being considered as a good compromise between cost and accuracy.
Authors
- Que Yang (ORCID: https://orcid.org/0009-0009-8944-089X)
- Yuang Hu
- Yukun Du
Institutions
- University of Chicago (US)
- Carnegie Mellon University (US)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-20
- DOI
- https://doi.org/10.21203/rs.3.rs-11785586/v1
- Primary Topic
- Privacy-Preserving Technologies in Data
- Type
- preprint