3D multimodal image registration for plant phenotyping

The use of multiple camera technologies in a combined multimodal monitoring system for plant phenotyping offers promising benefits. Compared to configurations that rely on a single camera technology, cross-modal patterns can be recorded that allow a more comprehensive assessment of plant phenotypes. However, the effective utilization of cross-modal patterns depends on image registration to achieve pixel-precise alignment - a challenge often complicated by parallax and occlusion effects inherent in plant canopy imaging. In this study, we propose a novel multimodal 3D image registration method that addresses these challenges by integrating depth information from a time-of-flight camera into the registration process. By leveraging depth data, our method mitigates parallax effects, facilitating more accurate pixel alignment across camera modalities. Additionally, we introduce an automated mechanism to identify and differentiate various types of occlusions, thereby minimizing registration errors. To evaluate the efficacy of our approach, we conduct experiments on a diverse dataset comprising six distinct plant species with varying leaf geometries. Our results demonstrate the robustness of the proposed registration algorithm, showcasing its ability to achieve accurate alignment across different plant types and camera compositions. Compared to previous methods our approach is not reliant on detecting plant-specific image features, making it suitable for a wide range of applications in plant sciences. Moreover, the registration approach can scale to arbitrary numbers of cameras with varying resolutions and wavelengths. Overall, our study contributes to advancing the field of plant phenotyping by offering a robust and reliable solution for multimodal image registration. • Novel multimodal image registration algorithm for plant phenotyping. • Utilizes 3D information from a depth camera and uses ray casting for registration. • Applicable for arbitrary multimodal camera setups and any plant species. • Integrated method to automatically detect and filter out occlusion effects. • Can compute both registered images and point clouds of plants.