A Unified Framework for Multimodal Joint Segmentation and Registration Combining Finite Distortion Theory and Implicit Neural Networks

Abstract Segmentation and registration are fundamental and interdependent steps in image analysis, particularly when dealing with multimodal data where matching salient structural information is challenging. Integrating both tasks within a single framework mitigates error propagation and fosters reciprocal reinforcement: registration provides topological priors that help disambiguate weak or missing boundaries and preserve contextual relationships between objects, while segmentation guides the alignment of anatomically or structurally relevant regions. In this line, we present an unsupervised model that merges variational formulations—valued for their balance between mathematical rigour and modelling flexibility—with neural approaches leveraging sinusoidal representation network, enabling efficient high-resolution processing. The method is grounded in finite distortion theory, providing a mathematically controlled deformation model that preserves homeomorphic structure. Furthermore, by promoting direction-agnostic alignment of image gradients via a directional total variation term, the method is naturally suited to heterogeneous modalities without requiring intensity matching. Although experimental validation is not the primary focus, the main contribution lies in the formulation itself, supported by theoretical results, including the existence of a minimiser for the derived optimisation problem. This work thus offers a proof of concept and establishes a new direction for unsupervised multimodal registration/segmentation methods. A preliminary version (Chan Sock Line et al., in: Bubba, Gaburro, Gazzola, Papafitsoros, Pereyra, Schönlieb (eds) Scale space and variational methods in computer vision, Springer, Cham, 2025) of this work has been published in the proceedings of the Tenth International Conference on Scale Space and Variational Methods in Computer Vision, 2025. It did not include the full theoretical results (on both directional total variation aspects and finite distortion mapping theory), nor the corresponding detailed proofs. Beyond these significant additions, the present version also provides a more comprehensive, systematic, and in-depth analysis of the numerical experiments, addressing several key aspects: (i) qualitative evaluation of the proposed joint model using multiple metrics, (ii) extension to multiphase and multimodal scenarios, and (iii) investigation of inverse consistency, among others.