A Multistep Registration Framework for Multimodal Histology Across Same and Consecutive Sections

Abstract Multimodal digital pathology combines hematoxylin and eosin (H&E) staining with advanced microscopy techniques to provide complementary structural and molecular information. However, accurate registration between H&E and other modalities remains challenging, especially when images differ strongly in appearance or are acquired on nonidentical consecutive sections. We present a multistep multimodal registration workflow that combines a multiscale histogram of local main orientations (MS-HLMO), a multimodal feature-based rigid initialization, with B-spline nonrigid registration and thin-plate spline refinement. We evaluated this approach on two datasets: human intestinal inflammatory bowel disease sections imaged with nonlinear multimodal microscopy (nMM) and H&E and head and neck cancer biopsies imaged with H&E and coherent Raman scattering (CRS), including single-band stimulated Raman scattering (SRS) and multimodal CRS images on parallel frozen sections. Registration performance was quantified using landmark and boundary-based metrics. The workflow was compared with several registration approaches, including scale-invariant feature transform (SIFT), B-spline deformable registration, MS-HLMO, and VALIS. For H&E/SRS pairs, the workflow reduced the median landmark error from 0.01459 to 0.01204 and lowered the mask boundary error from 0.0482 to 0.0115 compared with MS-HLMO. For H&E/nMM pairs, it achieved the lowest boundary errors, while B-spline registration yielded the smallest landmark errors, indicating a trade-off between point accuracy and boundary alignment. Qualitative checkerboards showed that the proposed method produced more consistent multimodal alignments than SIFT and VALIS across the evaluated datasets. These results demonstrate that the proposed workflow provides robust multimodal registration for both same-section and consecutive-section histology and enables reliable transfer of annotations between H&E and advanced microscopy modalities.