Optimizing autonomous multi-view stereo scans using AI based image masking within cultural heritage digitization

Image masking is essential in the field of 3D reconstruction for cultural heritage objects. It is used to accelerate the reconstruction process by removing background noise and accurately reconstructing the object only. The autonomous iterative Multi-View Stereo 3D scanner from the Fraunhofer Institute for Computer Graphics Research, within the Cultural Heritage Digitization department, requires binary masks to scan objects efficiently, regardless of the surrounding environment, geometry or color of the object, background color, and stabilizing mount. However, conventional masking methods can produce incorrect masks, leading to an inefficient or even abortive scan. Until now, these cases have been solved by parameter optimizations of the conventional masking method or changes in the scanning environment. This does not align with the principles of automation, since non-technical users in museums, archives, etc. should be able to use the autonomous iterative scanning workflow without additional effort. In addition to the real scan data used for training the presented networks, an automated Blender pipeline is also introduced, which generates additional synthetic data for training. Therefore, we evaluate if the latest state-of-the-art artificial intelligence segmentation methods can be used for these challenging cases without compromising their performance in simpler scenarios. This paper shows that with the proper network and datasets, masks of difficult objects or scenarios can be generated that can be used within the autonomous iterative scanning workflow. Thus, parameter and environment optimizations are no longer necessary.