Privacy distillation: Collaborative privacy protection for Transformer

Transformer have been widely used in the field of natural language processing due to their excellent performance, but the high computational cost and large parameter scale limit the efficiency of actual deployment. As an effective model compression technique, model distillation constructs small-scale student models from large models through knowledge transfer, significantly reducing resource requirements. However, there is a risk of information leakage during the distillation process of the model. This article deeply analyzes the potential mechanisms of information leakage in the distillation process of Transformer, including data reproduction attacks, reverse inference attacks, and other issues. In response to the above risks, a method combining differential privacy technology with secure multi-party computation was proposed for protection, and its effectiveness was verified through experiments. Research has shown that this method can significantly reduce the risk of information leakage while ensuring model performance.