Combining multimodal medical imaging and artificial intelligence for the early diagnosis of pancreatic cancer

1 Introduction Pancreatic cancer is a prevalent digestive system malignancy that poses a significant threat to human health. It ranks among the most lethal cancers, with an overall 5-year relative survival rate of approximately 11%(1). This disease is distinguished by nonspecific early symptoms, high invasiveness, high mortality, and low curability. In 2024, pancreatic cancer is projected to cause approximately 66,440 new cases and approximately 51,750 deaths, making it the third leading cause of cancer-related death, surpassing breast cancer(2). According to global cancer statistics, pancreatic cancer is projected to become the second leading cause of cancer-related death by 2030(3,4). The primary factor contributing to its high mortality rate is the lack of prominent early clinical symptoms, as well as the absence of definitive early diagnostic markers and effective screening strategies. Consequently, by the time of diagnosis, many patients have already progressed to advanced stages, missing the optimal treatment window(5,6). However, if pancreatic cancer could be accurately diagnosed in its early stages, both survival and cure rates could be substantially increased(7). Thus, the development of objective methods for the early, rapid, and precise diagnosis of pancreatic cancer remains a critical challenge.Currently, puncture biopsy remains the gold standard for diagnosing pancreatic cancer. However, it is invasive, costly, time-consuming, and associated with a high risk of complications such as infection, bleeding, and pancreatitis. Moreover, the heterogeneity of tumor tissues may compromise the representativeness of sampling, thereby limiting the accuracy of the test results. In recent years, the potential of noninvasive and comprehensive imaging techniques for the early diagnosis of pancreatic cancer has garnered significant research attention. Studies have demonstrated that multimodal medical imaging technologies, including endoscopic ultrasound (EUS), CT, and MRI, are widely applied during the preoperative evaluation of pancreatic cancer patients and have achieved promising results in early detection. Nevertheless, relying solely on imaging features for pancreatic cancer diagnosis heavily depends on physicians' visual assessments and diagnostic experience. This approach has several limitations, such as low sensitivity, atypical imaging characteristics of some pancreatic cancers, and susceptibility to misdiagnosis or missed diagnosis due to interobserver variability(8). With advancements in artificial intelligence (AI), AI-based models leveraging medical imaging technology have shown remarkable diagnostic accuracy and efficiency in assisting with pancreatic cancer diagnosis. By employing advanced algorithms to analyze large-scale medical image datasets, these models can rapidly and precisely identify subtle imaging features of pancreatic cancer that are imperceptible to the human eye, providing clinicians with reliable diagnostic references and significantly increasing the efficiency and quality of early pancreatic cancer diagnosis.In this work, we argue that, considering the critical importance of early pancreatic cancer diagnosis and the limitations of existing diagnostic methods, organically integrating multimodal medical imaging technologies—such as EUS, CT, and MRI—with distinct imaging principles to leverage their complementary strengths and comprehensively extract pancreatic cancer-related information represents a key breakthrough for achieving early and rapid diagnosis. Building on this foundation, developing an innovative AI-assisted framework for early pancreatic cancer diagnosis holds substantial importance. This framework should not only enable deep mining of potential key features embedded in multimodal image data but also increase diagnostic accuracy and stability through algorithm optimization. The ultimate aim is to provide clinicians with fast and reliable decision support, enabling them to make more efficient and accurate judgments in complex pancreatic cancer diagnostic scenarios while offering a promising direction for future advancements in the precise diagnosis of early-stage pancreatic cancer.2 Current status of artificial intelligence and multimodal medical imaging for the assisted diagnosis of pancreatic cancerEarly detection of pancreatic cancer remains an extremely challenging task. Owing to its high-resolution imaging ability, precise biopsy capability, and staging evaluation value, EUS has emerged as a critical tool for the early diagnosis, pathologic confirmation, and preoperative assessment of pancreatic cancer, particularly for small lesions, complex cases, or scenarios requiring minimally invasive interventions. Studies have demonstrated that EUS outperforms CT or MRI in diagnosing pancreatic cancer, with superior sensitivity for lesions ≤2 cm in diameter. Specifically, EUS can detect tumors as small as 5 mm in diameter, highlighting its significant advantages in this context(9,10). Recent research has indicated that AI can enhance the performance of EUS, with AI-assisted EUS models achieving diagnostic accuracy surpassing or matching that of human interpretation(11). A retrospective study(12) analyzed EUS images from 216 patients using a support vector machine (SVM) model to differentiate normal tissue from pancreatic cancer. The results revealed an accuracy of 98%, sensitivity of 94.3%, and specificity of 99.5%. Furthermore, a systematic review of 11 studies evaluating AI-assisted EUS modeling for pancreatic cancer diagnosis reported overall accuracies, sensitivities, and specificities ranging from 80–97.5%, 83–100%, and 50–99%, respectively(13). On the basis of current evidence, AI-assisted EUS models demonstrate promising potential for early pancreatic cancer detection, characterized by high diagnostic accuracy, despite being in the early stages of development and clinical application. For example, the Aichi Cancer Research Center in Japan developed a deep learning-based AI model capable of distinguishing pancreatic cancer lesions from noncancerous lesions using EUS images. In the validation set, this model achieved an area under the curve (AUC) of 0.90, a sensitivity of 0.94, and a specificity of 0.82(14).CT, which has superior spatial and temporal resolution, is widely recognized as the preferred noninvasive imaging modality for pancreatic cancer detection. It plays a critical role in diagnosing, staging, and evaluating treatment efficacy for patients with pancreatic cancer. Cao, K et al. introduced a deep learning framework named artificial intelligence for pancreatic cancer detection (PANDA), which was designed to detect and classify pancreatic lesions using noncontrast CT images. The model was trained on a single-center dataset of 3208 patients' noncontrast CT scans. In a multicenter validation involving 6239 patients from 10 centers, the model demonstrated exceptional performance, achieving an AUC ranging from 0.986 to 0.996 for lesion detection. This model is particularly advantageous for patients contraindicated for intravenous contrast(15). Mukherjee, S et al. developed a radiomic-based machine learning (ML) model for the prediagnostic detection of pancreatic cancer. Their study included prediagnostic CT scans from 155 patients with pancreatic cancer and 265 age-matched controls with a normal pancreas, and 34 imaging histologic features were selected. Validation was conducted on 176 in-house patients and 80 external controls. The SVM-based classifier achieved a sensitivity of 95.5% and an AUC of 0.98. These findings indicate promising results for early-stage detection, potentially improving outcomes by identifying tumors at a resectable stage(16). A study from Zhejiang University utilized abdominal-enhanced CT images from 319 patients to train a deep learning model capable of suggesting pancreatic tumor diagnoses on the basis of original abdominal CT images. The model achieved an AUC of 0.871 and an F1 score of 88.5%. Across all tumor types, the average diagnostic accuracy was 82.7%, with differential diagnostic accuracies of 100% for intraductal papillary mucinous neoplasms (IPMNs) and 87.6% for pancreatic ductal adenocarcinoma (PDAC)(17). Ma, H et al. screened 222 pathologically confirmed pancreatic cancer cases and 190 normal pancreas cases and trained a convolutional neural network (CNN) model for binary classification (presence or absence of pancreatic cancer) using 7245 CT images. The model exhibited an accuracy of 95.47%, sensitivity of 91.58%, and specificity of 98.27%, showing no significant difference compared with radiologists(18). In summary, artificial intelligence-assisted CT imaging for pancreatic cancer diagnosis has garnered substantial attention in recent years, demonstrating promising potential for increasing diagnostic accuracy and improving clinical decision-making.MRI is a widely used nonionizing radiation examination technique in clinical practice and is characterized by high soft tissue contrast and spatial resolution. It can more accurately reflect changes in tumor tissue components, such as the degree of fibrosis, microvessel density, hypoxia, and other alterations in tissue status and composition. Multiparameter quantitative analysis of tumor tissues has gradually become an essential auxiliary tool for the early diagnosis of pancreatic cancer(19). Combined multisquence MRI (anatomical and functional) is widely recognized as a critical tool for diagnosing, staging, and evaluating treatment efficacy in pancreatic cancer, demonstrating superior performance compared with CT in detecting small tumors and assessing vascular invasion. Li Qi et al. developed and validated an automated MRI-based model for preoperative differentiation between pancreatic squamous cell carcinoma and pancreatic ductal adenocarcinoma using conventional MRI and radiomic features, integrating clinical, radiomic, and hybrid models. The hybrid model combines MRI and radiomic features to distinguish pancreatic squamous cell carcinoma from pancreatic ductal adenocarcinoma(20). Additionally, deep learning-enhanced MR images, assisted by generative adversarial networks (GANs), have shown strong potential in discriminating pancreatic cancer from benign pancreatic diseases(21). Classifiers constructed on the basis of histogram arrays of MR images and CNNs were able to differentiate pancreatic cancer from pancreatic neuroendocrine tumors and solid pseudopapillary tumors, achieving AUCs of 0.896, 0.846, and 0.839 in the training, validation, and test groups(22), respectively. A recent study(23) further demonstrated that the integration of AI, hyperpolarized metabolic magnetic resonance (HP-MR), and multimodality imaging information may facilitate the development of real-time biomarkers for the early detection of pancreatic cancer, assessment of cancer aggressiveness, and early efficacy evaluation. However, HP-MR experiments are currently limited to preclinical models and have not yet been routinely applied in clinical settings.In summary, with the continuous advancement of artificial intelligence, an increasing number of studies have focused on diagnosing pancreatic cancer using AI-assisted images from various modalities, and some models have demonstrated promising diagnostic efficacy. However, current research remains limited to constructing diagnostic models based on single-modality images combined with artificial intelligence. Different imaging modalities possess distinct advantages and limitations. For example, the sensitivity and specificity of EUS are highly operator dependent. Less experienced endoscopists may fail to distinguish subtle imaging differences between early-stage pancreatic cancer and other pathologies(24–26). CT suffers from insufficient soft tissue resolution, radiation exposure, and contrast-related risks, whereas MRI is associated with long examination times, high costs, and numerous contraindications. Given the complexity of pancreatic cancer diagnosis and the inherent limitations of individual imaging modalities, clinicians must comprehensively evaluate multidimensional patient information, including pathological characteristics, baseline physical conditions, and economic affordability. They should carefully weigh the strengths and weaknesses of EUS, CT, MRI, and other imaging techniques to develop personalized and precise diagnostic plans, striving to obtain the most valuable diagnostic information with minimal medical risk. Multimodal medical images harbor rich information beyond mere morphological observations, reflecting not only lesion heterogeneity but also molecular features and prognosis-related data. In contrast, unimodal images inherently lack comprehensive information, limiting the amount of hidden image features that AI can extract. The synergistic integration of EUS, MRI, and CT provides a multidimensional approach to pancreatic cancer detection. EUS demonstrates exceptional sensitivity in identifying subcentimeter lesions, particularly for early-stage tumors undetectable by conventional imaging. Similarly, MRI offers superior soft-tissue contrast resolution, exemplified by sequences such as diffusion-weighted imaging (DWI) and MR cholangiopancreatography (MRCP), enabling precise characterization of parenchymal abnormalities and ductal involvement. Moreover, CT remains indispensable because of its rapid image acquisition, widespread availability, and superior performance in assessing local invasion and distant metastases. This trimodal strategy leverages the unique advantages of each technique, achieving diagnostic accuracy unattainable with any single modality alone. To address this, we propose an innovative strategy to integrate AI with multimodal images (EUS, CT, and MRI). This approach enables systematic mining of quantitative image features of pancreatic cancer across different modalities through high-throughput analysis to accurately elucidate the intrinsic connections between multimodal data and disease biological characteristics. On the basis of this integration, the intelligent diagnostic model for early pancreatic cancer detection, which is constructed by leveraging multimodal information, is expected to significantly enhance diagnostic performance and increase clinical application value.3 Our opinionTherefore, we propose that AI can utilize diverse types of multimodal image data from patients with pancreatic cancer for extensive AI model training. The aim of this approach is to provide a robust solution for the early diagnosis of pancreatic cancer by identifying noninvasive imaging biomarkers for early pancreatic cancer detection, constructing an end-to-end early screening system for pancreatic cancer based on multimodal images, and implementing a comprehensive intelligent visual diagnosis process from raw data input to clinical decision output.3.1 Synergistic Integration of Multimodal ImagesThe integration of a patient's EUS, CT, and MR images is a complex multimodal medical image processing task that requires a combination of image alignment, normalization, and fusion techniques. The first step is data preparation and preprocessing. This first step includes data format unification, which ensures that all the images are in the DICOM format (the standard format for medical imaging) or are converted to the DICOM format and checking whether the metadata are complete. Then, spatial resolution and orientation alignment, including resampling and orientation standardization, are performed. Finally, denoising and enhancement are performed. The second step is multimodal image alignment, which aligns images of different modalities to the same anatomical space and extracts anatomical landmarks, such as vascular bifurcations, for ultrasound images to correspond with CT/MRI. The third step is image normalization, which includes intensity normalization and spatial standardization. The fourth step is multimodal image fusion, including pixel-level fusion (displaying bones from CT and soft tissues from MRI superimposed on each other), feature-level fusion (extracting features of different modalities, such as calcified foci from CT and tumor boundaries from MRI, and then fusing them), and body data fusion. The final step is the verification of the fusion effect, in which the clinician evaluates whether the fused image meets the diagnostic needs, and the quantitative metrics include structural similarity (SSIM) and peak signal-to-noise ratio (PSNR). The process of integrating multimodal images must adhere strictly to ethical and privacy regulations.3.2 Building an End-to-End Diagnostic Model for Early Pancreatic Cancer Detection Using Artificial Intelligence and Multimodal ImagingWe propose that the integration of preprocessed and standardized multimodal data be used to construct an end-to-end early pancreatic cancer diagnostic model. The first step is multimodal data feature extraction, including both radiomic features extracted on the basis of manually delineated regions of interest (ROIs) and features automatically learned end-to-end by deep learning models. Radiomic features are extracted from standardized images using tools such as the PyRadiomics library, 3D Slicer, and ITK-SNAP. The types of extracted features include shape features, texture features, and intensity features. Deep learning automated feature mining requires designing separate branching networks for each modality, which are implemented using the feature extraction layer of the pretrained model. The second step is cross-modal dynamic fusion and feature alignment and splicing, where radiomic features and deep learning features are normalized and spliced into multimodal feature vectors. If the dimensionality of different modal features varies greatly, they can be mapped to a unified dimension through the full connectivity layer. The weights of different modal features are dynamically assigned to suppress redundant information. The third step is end-to-end model integration, where the model architecture requires multibranch inputs, and each modality is into the Then, the multimodal features are fused through the attention The step is model validation and and the evaluation metrics include the sensitivity, and well as the auxiliary and attention The process such as an insufficient amount of multimodal information between modalities, and limited can learning to the feature space alignment by contrast training, data to these the an end-to-end multimodal early pancreatic cancer diagnosis model with high can be of End-to-End for the Early of Pancreatic Cancer in propose that an end-to-end model for early pancreatic cancer diagnosis can be and into a clinical system for and application. For model can be on CT and MR images to the tumor regions of assisting in Additionally, 3D of the tumor can provide more anatomical information. A can be automatically key such as tumor of and risk of vascular invasion. To increase model the imaging features by the model can be demonstrated and compared with the diagnostic in clinical can also be to the and of each For integration with a based on or can be developed to DICOM images from the and input them into the model. can be developed for to the model results into the To data and and of data between the system and the model are can be model to and while all should also have the to and review the basis of model The clinical system can a malignancy with in the and providing to further or clinical validation of the model is A multicenter involving with several can be conducted to the sensitivity and specificity and its performance diagnoses by that in the it be to a for early pancreatic cancer diagnosis, enabling to leverage AI models the while to medical and in Early Pancreatic Cancer Using Artificial AI technology has demonstrated significant potential for application in the early diagnosis, and assessment of pancreatic cancer. However, its clinical application several and that to be of the significant of AI technology in the medical the of AI models remains and is as a can the input data and the data processing is The high accuracy of AI algorithms may at the of making it to enhance algorithm performance through to the Consequently, the of AI-assisted pancreatic cancer diagnostic models remains a critical factor limiting their widespread clinical and represents a key for future that the of artificial intelligence their lack of can be substantially through the integration of an attention and feature techniques to a The attention to precisely identify key anatomical regions of diagnostic importance at the For example, in CT the attention feature regions that the model such as and techniques the of deep neural the process from texture features to In pathological classification for example, this approach the for identifying in cancer This not only in but also provides that they with medical AI pancreatic cancer studies are single-center studies and small making them to and applied to other centers, these models in and leading to significant in accuracy and a lack of To address these it is essential to a framework and multicenter This approach ensures that the dataset is more thereby the and performance of AI models. Additionally, various data algorithms can be to these and increase the and of raw advancements in the application of artificial intelligence models for pancreatic cancer diagnosis, numerous in their clinical These include a of retrospective of and and diagnostic and Additionally, the lack of standardized for evaluating diagnostic accuracy poses a significant Furthermore, diagnostic developed by different across various studies a high risk of and an the ethical associated with AI must not be The of data should strictly adhere to the of involving both and which to some the application of AI In many pancreatic cancer data are and the process is However, AI is an and system that requires continuous of data from new clinical These new patients their data to the AI system it is used for diagnosis and Finally, is AI diagnostic which in and clinical the for AI model development is in such To address this in the and the application of AI and from existing ethical and for medical AI is For example, a examination of the and AI to patient data or an analysis of automated the to and to scenarios of medical AI misdiagnosis are By from these a solid and a more comprehensive can be for the boundaries of and designing resolution as research on AI technology has has been a to the diagnosis and treatment for early-stage pancreatic cancer. For future we propose that AI can be with multimodal medical imaging to develop an end-to-end automated early pancreatic cancer screening This facilitate between the system and the screening enabling intelligent raw data input to clinical Consequently, this approach pancreatic cancer to be and at its stages, providing a promising to its high mortality