Privacy-preserving transfer learning via one-time encrypted data filtering

Private neural network training on encrypted data provides strong protection for user identity privacy. However, its substantial computational overhead hinders practical deployment, particularly on large-scale datasets where full training cycles may require thousands of hours. Current FHE-based privacy-preserving approaches predominantly rely on transfer learning, which appends and fine-tunes new classification layers atop pre-trained models. Nevertheless, transfer learning often introduces significant redundancy when applied to new datasets, considerably prolonging the encrypted training process. To mitigate this, we propose a simple yet efficient FHE-compatible algorithm for sample importance evaluation. ​​A core innovation is that sample scoring is performed exclusively during the initial training epoch​​, eliminating repeated full-dataset evaluations throughout the entire training cycle and drastically reducing computational overhead. ​​Furthermore, our evaluation method entirely avoids polynomial homomorphic evaluations​​, circumventing a major source of latency in existing FHE-based methods. Building on this, we design a sample reorganization strategy that leverages a carefully initialized encrypted data layout to rapidly assemble informative samples and prune redundant ones, thereby substantially reducing the effective dataset size. Experimental results demonstrate that our approach achieves a 30% speedup in training time compared to state-of-the-art methods, with only a marginal accuracy drop of 0.6%, highlighting its practical utility for privacy-preserving machine learning.