Deconstruct to reconstruct: an automated pipeline for parsing complex CT assemblies

Abstract Many technical products are assemblies formed from smaller, versatile building blocks. Deconstructing such assemblies is an industrially important problem and an inspiring challenge for machine learning approaches. For the first time, we present an effective and fully automated pipeline for parsing large-scale, complex 3D assemblies from computed tomography (CT) scans into their individual parts. We have generated and make available a high-quality dataset of simulated, physically accurate CT scans with ground truth annotations. It consists of seven high-resolution CT scans ( $$\sim \! 2000^3$$ voxels) of different technical assemblies with up to 3600 parts, each annotated with instance and semantic labels. The parts strongly vary in size and sometimes differ in fine details only. Our pipeline successfully handles the high-resolution volumetric inputs (3–30 GB) and produces detailed reconstructions of complex assemblies. The pipeline combines a 3D deep boundary detection network trained only on simulated CT scans with efficient graph partitioning to segment the 3D scans. The predicted instance segments are matched and aligned with a known part catalog to form a set of candidate part poses. The subset of these proposals that jointly best reconstructs the assembly is found by solving an instance of the maximum weighted independent set problem. We demonstrate that our approach generalizes to different CT scan setups and yields promising results even on real CT scans. Our pipeline is applicable to models that include parts not seen during training, making our approach adaptable to real-world scenarios.