Automated Multimodal Image Registration for Prostate Cancer Using Squeeze-and-Excitation ResNet with Thin Plate Spline Transformation: A Deep Learning Approach

BACKGROUND Accurate spatial correlation between preoperative prostate MRI and post-prostatectomy histopathology is critical for improving prostate cancer diagnosis, treatment planning, and MRI interpretation. Current manual registration methods are time-consuming and subjective, creating a need for robust, automated solutions. MATERIAL AND METHODS We developed an unsupervised deep learning model, Squeeze-and-Excitation ResNet with Thin-Plate Spline Transformation (SE-ResNet-TPS), for deformable registration of in vivo prostate MRI and ex vivo whole-mount histopathology images. The model learns feature correspondence directly from unlabeled image pairs, eliminating dependency on large annotated datasets. It integrates multi-scale convolutional kernels within a ResNet architecture and incorporates a channel attention mechanism (Squeeze-and-Excitation) to enhance sensitivity to diagnostically relevant features. A thin-plate spline (TPS) transformation module is employed to model complex global and local deformations between the inherently different modalities. RESULTS The SE-ResNet-TPS model was rigorously evaluated. It achieved an overall Dice similarity coefficient (DSC) of 0.964 and a Hausdorff distance (HD) of 2.91, indicating excellent anatomical alignment between the registered MRI and histopathology images. For cancer-specific regions of interest, where registration is most challenging, the model yielded a DSC of 0.578 and an HD of 4.97, demonstrating significant capability in aligning clinically critical areas despite modality differences and tissue processing artifacts. CONCLUSIONS The proposed SE-ResNet-TPS framework provides highly accurate, unsupervised registration of prostate MRI and histopathology images. Its performance, particularly in aligning overall anatomy, confirms its effectiveness for multimodal prostate image fusion. While cancer-specific alignment presents greater challenges, the results are promising. This model has strong potential to enhance the precision of prostate cancer localization on MRI, ultimately supporting radiologists in diagnosis and targeted biopsy guidance.