Combined Self-Supervised Learning for ECG Classification Based on the Classification Token

Electrocardiogram (ECG) is a crucial non-invasive method for measuring the electrical activities of the heart and detecting cardiovascular diseases. While deep learning approaches for cardiovascular disease classification have gained popularity, creating labeled data remains expensive. Contrastive and generative self-supervised learning methods, particularly with the vision transformer (ViT) neural network architecture, have been introduced as leading solutions to this challenge. Although the [CLS] token, which is one of the most important components of ViT, is frequently utilized for aggregate representation in supervised learning scenarios, such as diagnosis, its exploitation in self-supervised tasks has not been extensively explored. This study proposed a method to incorporate multiple pretraining tasks for better representation learning via utilizing the [CLS] token more effectively. Based on this method, we introduced two novel combined self-supervised learning frameworks for ECG analysis, namely MAE-MoCo and MAE-Nextclip. The MAE-MoCo framework combines generative and contrastive self-supervised learning by incorporating a masked auto encoder with a momentum encoder. On the other hand, MAE-Nextclip is a generative method that reconstructs not only masked patches but also Nextclip data with the assistance of the [CLS] token. We validated our methods on a joint database of China Physiological Signal Challenge in 2018, Physikalisch-Technische Bundesanstalt XL, and Chapman datasets. The fine-tuned models outperformed the state-of-the-art models in the Chapman dataset with macro-Fl of 0.96 and the area under the curve of receiver operating characteristic of 0.99. The outstanding performance on downstream tasks demonstrates the potential of combining pretraining tasks, especially generative and contrastive tasks, in the field of automatic ECG interpretation.