Improving ViT Performance for Colon Cancer Histopathological Image Classification Using Transfer Learning

Deep learning has revolutionized medical image analysis, particularly the automatic classification of histopathology images. Vision Transformers (ViTs) now surpass convolutional neural networks (CNNs) in modeling long-range dependencies and capturing global features. However, when initialized with random weights, ViTs often struggle to adapt to the varying scales and morphologies of tumor tissues, resulting in suboptimal performance. Transfer learning addresses this challenge by pretraining the model on large datasets to obtain a stronger weight initialization before fine-tuning on the target domain. This strategy has proven especially effective for histology image classification. In this study, we fine-tuned a ViT-B/16 model pretrained on ImageNet-21k by first training it on 75,000 Renal Cell Carcinoma (RCC) patches and then applying it to classify colon tumor subtypes in the Chaoyang dataset. Our proposed method achieves an accuracy of 87.14 %, a precision of 83.59 %, a recall of 82.08 % and an F1 score of 82.77 %, outperforming both a model trained from scratch and one pretrained only on ImageNet. These results underscore the value of domain-specific pretraining for enhancing ViT performance on diverse tumor morphologies in histopathology classification tasks.