Transformer-Based Self-Supervised Learning and Distillation for Medical Image Classification: Improving Colorectal Cancer Detection on NCT-CRC-HE-100K with Swin-T V2

In this work, we present a novel approach to colorectal cancer tissue classification using Swin-Transformer V2 on the NCT-CRC-HE-100K dataset. This study is the first to apply Swin-Transformer V2 in this domain, leveraging its advanced architecture to achieve state-of-the-art performance. Building upon the success of previous self-supervised learning methods like MoBY, which demonstrated strong results on natural images, we extend these techniques to medical datasets. We perform self-supervised pretraining on a wide range of tumor-related datasets, incorporating advanced data augmentation strategies, such as random cropping and 2x magnification, to address the multi-scale nature of histopathological images. After pretraining, we employ a progressive layer-wise distillation technique, transferring knowledge from a large teacher model to a more efficient student model. This method dynamically adjusts the distillation strength across layers using a learnable parameter, improving training efficiency and overall performance. Our approach achieves a top-l accuracy of 96.0% on NCT-CRC-HE-100K, surpassing the previous best by 0.5 percentage points. This work demonstrates the potential of Transformer-based architectures in medical imaging tasks and highlights the effectiveness of self-supervised learning and distillation techniques in improving model accuracy.