A dynamic visual localization method for intelligent coal gangue sorting robots

Abstract During dynamic grasping by coal gangue sorting robots, conveyor speed fluctuations, belt slippage, belt deviation, and vibration can cause deviations between the predicted and actual target positions, leading to empty grasps, missed grasps, and grasping failures. To address the difficulty of fixed-speed prediction and encoder-based displacement estimation in perceiving target-state changes in the sorting region in real time, a dynamic visual localization method based on a hand-eye vision system is proposed. The proposed method uses visual tracking as the primary localization mode. Based on the SiamDW framework, multi-scale features and a lightweight channel attention mechanism are integrated to construct the SiamCSF localization model, enhancing target feature representation under scale variation, camera-target distance variation, and background interference. Meanwhile, the average peak-to-correlation energy is used to evaluate visual localization reliability. When visual confidence decreases, encoder displacement is used for short-term compensation along the conveying direction, with the previous visual localization result serving as the reference state, thereby improving localization continuity and stability during dynamic sorting. Experimental results show that the proposed method achieves stable localization under different conveyor speeds, camera-target distance variations, and visual feature degradation conditions. It achieves a localization frame rate of about 58 FPS, with average localization errors of approximately 1.7 mm and 1.8 mm in the X and Y directions, respectively, satisfying the real-time and accuracy requirements of dynamic grasping.