A PET/CT Dual-modal Breast Cancer Classification Method Based on Vision Transformer

Breast cancer has become the most prevalent and lethal disease among women worldwide. Over the past years, Positron Emission Tomography/Computed Tomography (PET/CT) has emerged as an indispensable modality in breast cancer diagnosis. However, image data from a single modality often fails to comprehensively reflect the characteristics of lesions, which significantly restricts the accuracy and reliability of diagnosis. To find a solution for this complex challenge, this paper proposes a Multi-modality Medical Vision Transformer (MM-ViT). The primary objective of MM-ViT is to conduct an integrated analysis of PET/CT dual-modal data, aiming to achieve more precise breast cancer subtyping solely using medical imaging. The effectiveness of MM-ViT was validated using 3,797 pairs of PET/CT images from 271 patients with four breast cancer subtypes. The influence of different baseline networks and feature fusion methods on classification performance was analyzed through extensive ablation and comparative experiments. The experimental results indicate that the proposed MM-ViT outperforms baseline models and existing methods. A four-class classification accuracy of 94.52% was achieved. This work provides new insights for personalized breast cancer treatment and demonstrates significant clinical potential.