Gradient Inversion of Text-Modal Data in Distributed Learning

Gradient inversion attacks (GIAs) pose significant challenges to the privacy-preserving paradigm of distributed learning. These attacks employ carefully designed strategies to reconstruct victim’s private training data from their shared gradients. However, existing work mainly focuses on attacks and defenses for image-modal data, while the study for text-modal data remains scarce. Furthermore, the performance of the limited attack researches on text-modal data is also unsatisfactory, which can be partially attributed to the finer granularity of text data compared to image. To bridge the existing research gap, we propose a high-fidelity attack method tailored for Transformer-based language models (LMs). In our method, we initially reconstruct the label space of the victim’s training data by leveraging the characteristics of the Transformer architecture. After that, we propose a shallow-to-deep paradigm to facilitate gradient matching, which can significantly improve the attack performance. Furthermore, we develop a weighted surrogate loss that resolves the consistent deviation issue present in current attack researches. A substantial number of experiments on Transformer-based LMs (e.g., Bert and GPT) demonstrate that our attack is competitive and significantly outperforms existing methods. In the final part of this paper, we investigate the influence of the inherent position embedding module within the Transformer architecture on attack performance, and based on the analysis results, we propose a countermeasure to alleviate part of the privacy leakage issue in distributed learning.