Efficient Multi-Stage Self-Supervised Learning for Pathology Image Analysis via Masking

Annotated large-scale datasets are crucial for pathology image analysis, yet creating such datasets is challenging. Self-supervised learning (SSL) offers a potential solution to reduce the need for extensive annotations. However, applying SSL methods designed for natural images directly to pathology images may not yield optimal efficiency and performance due to the unique characteristics of pathological images. In this paper, we explore the strengths and weaknesses of two widely used SSL categories, including generation-based SSL and contrastive SSL in pathology tasks. Our findings indicate that pathology images exhibit a high degree of information redundancy, making the direct application of these SSL categories inefficient and suboptimal. To address this, we propose the Rapid Histopathology Pre-training (RHP) framework, which employs a simple yet effective method for training vision Transformers. RHP uses a dual-branch random masking strategy to integrate the characteristics of pathological images with the advantages of both self-supervised schemes. Additionally, an extra multi-stage strategy can enhance model performance while mitigating domain discrepancy issues. This approach enables rapid training on pathological images, yielding superior performance compared to baseline methods. For our experiments, we extracted 4,821,041 multi-scale pathological images from 37,378 pathology slides for pre-training. We evaluated RHP’s performance on four downstream tasks, including patch classification, nucleus segmentation, gland segmentation, and whole slide image classification across eight public datasets. The results demonstrate that our RHP-based visual representation achieved state-of-the-art performance on each dataset with high efficiency, proving to be more robust compared to other SSL methods and ImageNet transfer learning. Our code is publicly released on GitHub (<uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/AndrewTal/RHP</uri>).