LEARNING GENERALIZABLE DEFORMATIONS FROM IMAGES

Medical image registration aims to estimate the spatial transformation between two medical images. It plays a crucial role in various medical applications, including longitudinal studies, radiation treatments, image-based biopsies, and brain studies. The rise of deep learning has led to a substantial number of studies on learning-based registration methods, where neural networks are trained to predict the transformation given a single pair of images as input. Over the past decade, these learning-based methods have demonstrated improved accuracy and faster run-time compared to conventional registration methods that rely on numerical optimization to seek the solution. However, these methods still encounter challenges in generalizing to real-world medical problems, which limits their practical applicability despite their superior accuracy and speed. My thesis aims to explore the nature of the difficulty of generalizing a registration neural network across different tasks, for high-resolution image pairs, and for challenging scenarios where the correspondence is ambiguous; and to develop practical methods that mitigate these difficulties. To enhance the generalizability of learning-based registration methods across various registration tasks, I pursued two directions. I investigated the reasons behind the difficulty of current learning-based registration methods in generalizing across different tasks. I proposed GradICON and uniGradICON, unlocking the possibility of developing a universal registration network that provides accurate registration results across registration tasks without retraining. In addition, I investigated SAME++ to combine pre-trained self-supervised representations with conventional registration methods to leverage the strengths of both approaches: accurate, fast runtime from learning-based methods and generalization from conventional methods, improving the registration accuracy for conventional methods. Apart from the challenges related to generalizing across different registration tasks, learning-based registration methods also encounter difficulties when applied to high-resolution images. The memory usage of training a 3D registration neural network increases cubically with the resolution of the input images, restricting the application of the registration methods to increasing resolutions, either in the form of greater image shapes or smaller spacing of the voxels. To ad- address this issue, I introduced NePhi, which significantly reduces memory consumption during training while preserving registration accuracy and runtime. In addition to 3D to 3D registration, which is commonly used for longitudinal or brain studies, 2D to 3D registration poses another challenging problem, particularly in the development of image-guided biopsy systems. The 2D to 3D registration problem often suffers from spatial ambiguity, making it difficult to directly apply the learning-based methods proposed for 3D to 3D registration problems. I investigated the registration between pre-operative 3D CT images and intra-operative 2D radiographs and proposed a differentiable rendering operation to transform the 3D CT images into 2D projections, facilitating an optimization-based solution for 2D/3D registration. Additionally, I developed LiftReg, a learning-based 2D/3D registration network capable of estimating inspiration/expiration motion in real-time.