A Coarse-to-Fine Multimodal Detection Framework Based on Deep Learning for Robotic Coating Tasks

Automated robotic systems are evolving rapidly in industrial manufacturing. In robotic seal coating scenarios, while RGB-D based perception is adaptable for flexible demands, it struggles with achieving high accuracy. To address this limitation, we propose a multimodal visual detection system for robotic seam gluing, which employs a coarse-to-fine strategy across two- and three-dimensional (2-D and 3-D) data modalities. The proposed framework integrates two main components: a 2-D seam segmentation & classification network for coarse seam identification at the image level, and a 3-D seam refinement & transformation network (3D-SRTN) for precise seam refinement at the point cloud level. Specifically, 3D-SRTN performs local seam point optimization, followed with global transformation to minimize cumulative coordinate errors. A postprocessing procedure is then designed to generate smooth 6-DoF paths. Additionally, we introduce the RGB-D coating dataset, specifically tailored for this task, which includes both 2-D and 3-D annotated data for RGB-D coating perception. Ablation studies validate the effectiveness of individual modules, while comparative experiments demonstrate the overall superiority of our system.