Pancreas Segmentation with Multi-Phase Feature Aggregation and Modality Adaptive Transformer

Automatic pancreas segmentation can facilitate diagnosis and treatment of pancreatic diseases. The combination of non-contrast, arterial, and venous phases of CT imaging can enhance differentiation of the pancreas from its surrounding structures. However, existing multi modal methods, which try to integrate the multimodal in formation in computer-aided pancreas segmentation, often overlook the inter-modal relationships and have a limited capability for information fusion. In this paper, we propose a multi-phase pancreas segmentation method for incorporating Feature Aggregation Module (FAM) and Modality Adaptive Transformer (MAT). Specifically, we use the venous phase as the primary modality, while the non-contrast and arterial phases serve as supplementary modalities, based on clinical prior knowledge. Our FAM integrates spatial information from the primary and supplementary modalities, while our MAT adaptively enhances feature representation and establishes long-range dependencies among modalities. Our method outperforms state-of-the art techniques on a large scale dataset. Based on the segmented pancreas region, We further perform a down-stream task focused on pancreatic volume calculation. The prediction accuracy is on par with manual segmentation, demonstrating effectiveness and potential application of our proposed method.