Enhancing Noncontact Vibration Monitoring With mmWave Radar and Camera Fusion

Automated manufacturing is the cornerstone of the Industrial Internet of Things (IIoT) ecosystem, where vibration monitoring technology is a critical tool for maintaining industrial machinery. The prevailing approach mostly employs inertial measurement units (IMUs), lasers, and cameras, each demonstrating deployment constraints. In recent years, millimeter-wave (mmWave) radar has shown high vibration measurement performance, but it faces challenges in accurately localizing vibrating objects and determining observation points. This study introduces a new system called VibCamera, which leverages the mmWave vibration measurement technology with computer vision (CV) algorithms for vibration monitoring. With the positional assistant of CV semantic segmentation, the radar can accurately determine sufficient observation points, thereby achieving precise measurement with high directionality. VibCamera includes two camera modes, RGB-only and RGB+depth, and solves two technical challenges: 1) integrating multimodal information for vibration target localization and 2) extracting high-quality vibration signals in interference environments. VibCamera provides more consistent and precise outcomes without the need for physical contact. The experimental results indicate that the RGB-only mode has amplitude and frequency errors below <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$27.04 \; \mu \rm m$ </tex-math></inline-formula> and 0.22 Hz, respectively, with a 90% probability, and the RGB+depth mode has errors below <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$23.72 \; \mu \rm m$ </tex-math></inline-formula> and 0.21 Hz.