Approaching Expressive and Secure Vertical Federated Learning With Embedding Alignment in Intelligent IoT Systems

In the context of vertical federated learning (VFL), agents utilize multimodal data on their edge devices to corporately train and inference with the deep learning models. However, in classical VFL, there exists three problems from the perspective of embeddings. 1) the utilization of oversimplified embedding fusion mechanism may result in suboptimal performance of the models; 2) the exchange of embeddings and their gradients poses a potential risk of private information leakage, as they inherently contain sensitive information; 3) finally, the withdrawal of some agents from cooperation disrupts the collaborative inference capabilities of the remaining agents. To mitigate these problems, this article introduces a novel VFL algorithm grounded in embedding alignment. It includes two distinct schemes: 1) performance-oriented scheme (POS) and 2) privacy-respecting scheme (PRS). Within POS, this article employs contrastive loss and joint fine-tuning to augment the expressiveness and the overall performance of models. While the PRS incorporates homomorphic-encryption-based contrastive loss and individual fine-tuning to safeguard the data security. In addition, the PRS eliminates the necessity of collaborative inference. In this article, comprehensive security analysis and proofs are conducted for PRS. Moreover, experiments demonstrate the superior performance of the proposed POS over classical VFL, showcasing a substantial performance improvement. Simultaneously, the PRS surpasses the performance of training alone, even under stringent security constraints.