Secformer: Privacy-preserving atomic-level componentized transformer-like model with MPC

The global surge in Artificial Intelligence (AI) has been triggered by the impressive performance of deep-learning models based on the Transformer architecture. However, the efficacy of such models is increasingly dependent on the volume and quality of data. Data are often distributed across institutions and companies, making cross-organizational data transfer vulnerable to privacy breaches and subject to privacy laws and trade secret regulations. These privacy and security concerns continue to pose major challenges to collaborative training and inference in multi-source data environments. These challenges are particularly significant for Transformer models, where the complex internal encryption computations drastically reduce computational efficiency, ultimately threatening the model's practical applicability. We hence introduce Secformer, an innovative architecture specifically designed to protect the privacy of Transformer-like models. Secformer separates the encoder and decoder modules, enabling the decomposition of computation flows in Transformer-like models and their efficient mapping to Multi-Party Computation (MPC) protocols. This design effectively addresses privacy leakage issues during the collaborative computation process of Transformer models. To prevent performance degradation caused by encrypted attention modules, we propose a modular design strategy that optimizes high-level components by reconstructing low-level operators. We further analyze the security of Secformer's core components, presenting security definitions and formal proofs. We construct a library of fundamental operators and core modules using atomic-level component designs as the basic building blocks for encoders and decoders. Moreover, these components can serve as foundational operators for other Transformer-like models. Extensive experimental evaluations demonstrate Secformer's excellent performance while preserving privacy and offering universal adaptability for Transformer-like models.