Tackling Heterogeneity in Intelligent Internet of Medical Things Applications Through Federated Transformer Regularization

In the conventional Internet of Medical Things (IoMT) system, patient data shared by the IoMT devices are susceptible and confidential and can be exposed to the antagonist. Hence, federated learning (FL) has become a promising solution for preserving the privacy of patient data. However, various complex variances in the IoMT context, such as device and data heterogeneity, model complexities, and others, may degrade the performance of the traditional FL model. This paper proposes a transformer-based federated learning framework for the IoMT network within a cloud-edge architecture. The proposed framework mitigates data and device heterogeneity in the IoMT framework by learning the IoMT devices through a multi-head self-attention (MHSA) mechanism. The transformer model is applied to each device’s local data to capture the complex pattern from each input data. Each device uses the transformer model as a part of its local model to minimize the local loss function. To reduce the risk of over-fitting in the transformer model, a regularization term is instigated to the local loss function of the learning process. Furthermore, it is shown that the proposed method augmented with the regularization factor converges to a stationary point θ* with a convergence rate of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</i>(1/<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</i>) in some situations. In addition, the proposed method provides ϵ-privacy guarantees on a local data set of IoMT devices. The proposed framework is investigated on the Human Activity Recognition (HAR), Physical Activity Monitoring (PAMAP2), and Physikalisch-Technische Bundesanstalt (PTB)-XL datasets. It measures the performance of the proposed methods compared to the state-of-the-art on various parameters.