RGB-D multimodal fusion perception-to-action system for automated detection and robotic sorting of construction and demolition waste

The integration of computer vision, grasp planning, and robotic execution enables automated sorting of construction and demolition (C&D) waste in cluttered environments. However, dust interference, depth noise, and spatial-temporal misalignment in dynamic scenes degrade sensing reliability, causing sample imbalance, unstable grasp prediction, spatial localization deviation, and redundant grasp execution. This paper proposes an RGB-D multimodal fusion perception-to-action system for detection, spatial localization, and robotic execution. A depth-aware preprocessing pipeline and adaptive denoising strategy improve feature quality, while focal loss addresses sample imbalance. A detection information selection mechanism refines outputs and suppresses redundant grasp commands, enhancing decision reliability. Inference efficiency is improved via decorator-based optimization. The system achieves real-time performance at 33.0 FPS, with a mean grasp angle error of 2.64° and an overall accuracy of 95.24%, outperforming the baseline by 33.34%. Robotic experiments show over 97% recognition accuracy and a weighted sorting effectiveness index of 95%, validating reliable perception-to-action integration.