Transmednet: Transformer Medical Triad Neurology Networks

ABSTRACT In recent years, Transformers have gradually gained widespread application in the field of computer vision (CV). However, when applied to medical imaging, common slicing strategies often miss key information along the third dimension. To address this, we propose a novel diagnosis model, Transformer Medical Triad Neurology Networks (TransmedNet), designed to better capture 3D brain image features. The technological innovations of this model are primarily manifested in three aspects: Firstly, the model employs a hierarchical mechanism. The bottommost layer partitions the brain, enhancing the comprehension within each brain region. The intermediate layer employs a segmentation moving window mechanism to extract correlations between adjacent windows. The topmost layer utilizes global multi‐head self‐attention to focus on overall correlations. Secondly, the model adopts a combination of Transformer and convolutional neural network architectures to balance global and local features, enhancing the overall performance of the model. Lastly, the model thoroughly considers the three‐dimensional features of brain images by incorporating a three‐dimensional multi‐head self‐attention mechanism, ensuring equal importance is given to each dimension. Our experiments yielded promising results. The classification accuracy reached 99.21% for distinguishing Alzheimer's disease (AD) from cognitively normal (CN) subjects, and 97.46% for distinguishing autism spectrum disorder (ASD) from CN. The results demonstrate that the TransmedNet model enhances the classification performance of brain imaging diseases.