Automatic detection and classification of sub-parts in the sub-assembly process of ship structures using depth image processing and deep learning

• We develop an automated detection and classification system for sub-parts in shipyard sub-assembly processes. • A RANSAC-based plane estimation method robustly extracts the topmost sub-part under stacking and tilting conditions. • The proposed system was quantitatively evaluated through field experiments, achieving an average pose error of 0.10% and an average distance error of 0.9%, thereby demonstrating reliable performance • The proposed system enables reliable and practical measurement automation in real shipyard operations. To automate the arrangement of structural parts in shipyards, it is essential to develop a technique capable of rapidly and reliably recognizing parts even under complex on-site conditions such as stacking or tilting. In this study, we developed an automated system that accurately separates and recognizes the topmost area of a part, precisely extracts its contours and hole regions, and classifies the type of part by utilizing 3D point clouds acquired from a depth camera in the working environment. The proposed RANdom SAmple Consensus (RANSAC)-based algorithm determines the optimal plane and recognizes the part by comprehensively considering the direction of the normal vector, distribution of inliers, and inclusion of the topmost area. Subsequently, the inlier data obtained from RANSAC are converted into a binarized image and fed into the YOLOv8 model to classify the part number. The proposed system was quantitatively evaluated through field experiments and exhibited reliable performance in that it achieved an average pose error of 0.123° (0.22%) and an average dimensional error of approximately 12 mm (0.73%). These results demonstrate the potential of the proposed system as a practical measurement system to extract geometric features and estimate an optimal pose of structural parts in complex shipyard environments.