Toward Accurate Vertical Federated Learning via Early Exit and Election Knowledge Distillation

As a promising privacy-aware collaborative machine learning framework, Vertical Federated Learning (VFL) enables knowledge sharing among various data owners with overlapping data samples but different features. Although VFL can form a global model for all data owners to enhance their inference performance, existing VFL methods suffer from the problem of low training performance. During the training processes of existing VFL methods, due to the interference between local models caused by the differences in local data features, the overall inference potential of the global model is greatly suppressed. To address this issue, this paper presents a novel VFL approach named VFL-EE based on early exit layers and our proposed election distillation scheme. By introducing one personalized early exit layer for each party on the server, VFL-EE can prevent one local model from being adversely affected by other models, thus effectively ensuring the personalization of each local model. Meanwhile, by employing our proposed minimum loss-based election distillation strategy, VFL-EE allows indirect knowledge sharing among local models, which further enhances the inference capabilities of the global model without compromising the personalization of local models. In this way, each party can achieve a personalized local model specifically tailored for the feature extraction of its local data.Comprehensive experiments on well-known datasets show that VFL-EE significantly outperforms existing VFL methods and can even improve the overall inference accuracy by up to 12%.