SeqFRT: Towards Effective Adaption of Foundation Model via Sequence Feature Reconstruction in Computational Pathology

Given the intricate situation of modelling gigapixel images, the usage of multiple instance learning (MIL) framework has recently increased to support clinical practice, encompassing cancer diagnosis, subtyping, survival prediction and other tasks. In current practice, most state-of-the-art MIL proposals typically apply a frozen pre-trained CNN or a pathological foundation model for feature extraction. While this paradigm lacks the capability for sequence feature fine-tuning within the downstream-specific tasks, which hinders the continuous performance promotion in Whole Slide Images (WSIs) Analysis. To address this issue, we propose a Sequence Feature Reconstruction Transformer (SeqFRT) for optimizing feature extraction of the foundation model, which can capture more discriminative features within pathological instance sequences. The proposed model comprises three main modules: 1) an offline foundation model as the pathological feature extractor; 2) a sequence position optimization architecture which aims at refining the correlations between instances in both sequential ordering and transpositional ordering; 3) a sequence sparsity enhancement strategy is designed to reconstruct the sequence feature and extract the latent representations instead of redundant information. Extensive experiments on six benchmark datasets for three computational pathology tasks demonstrated our model’s superiority over the state-of-the-art MIL methods. The source code is available at https://github.com/caicai2526/SeqFRT-MIL.