Possibility of Using the Attention Mechanism in Multimodal Recognition of Cardiovascular Diseases

The paper is about studying the possibility of using the attention mechanism in diagnosing various cardiovascular diseases. Biomedical data were presented in different modalities (text, images, and time series). A comparison of the efficiency of 5 transformers based on the attention mechanism (Dosovitsky transformer, compact convolutional trans­former, transformer with external attention, transformer based on tokenization with patch shift and local self-attention, transformer based on multiple deep attention) was carried out with the Exception convolutional neural network, three fully connected neural networks (MLP-Mixer, Fnet, and gMLP), and the YOLO architecture on the problem of multi-class classification (16 classes of dangerous arrhythmias). High efficiency of transformers in diagnosing cardiac diseases was shown. The transformer based on tokenization with patch shift and local self-attention showed the greatest efficiency.