Semantic 3D product modelling for automated inspection in remanufacturing processes

Abstract Remanufacturing is a key process for enabling a circular economy by restoring used products, often referred to as cores, to a like-new condition. It still heavily relies on manual work. One exemplary manual task is the visual inspection of cores before further processing. This manual effort arises from uncertainties, such as varying product conditions, a broad variety of product variants, and the lack of product information to support automation. To address these challenges and to enable the automation of visual inspection tasks, flexible and adaptive inspection systems are required. These systems must be capable of automatically detecting defects and performing product-specific inspections across a wide range of product variants. Therefore, this work introduces a semantic 3D product modelling method that integrates two-dimensional (images) and three-dimensional (point cloud) data. With the resulting semantic 3D model, the semantic information, such as detected defects and components requiring closer inspection, can be encoded on a geometric product model. This model provides the information basis for an adaptive inspection approach. Using semantic (U-Net) and instance segmentation approaches (Mask-RCNN), the proposed method assigns each surface point of the 3D model to a specific component, thereby creating the semantic 3D product model during the inspection process. The results show that the method presented can achieve semantic 3D modelling both in accuracy and model completeness, encoding component information on the geometric product model. Furthermore, we show that the U-Net architecture used to detect components is also able to detect corrosion as one exemplary defect type, enabling the encoding of various semantic information into the semantic 3D product model. This semantic 3D product model then enables targeted individual inspection of these semantically mapped components and defects on the product model in a later stage of an automated inspection procedure.