Seeing the Invisible: Recovering Surveillance Video With COTS mmWave Radar

Video surveillance systems play a crucial role in ensuring public safety and security by capturing and monitoring critical events in various areas. However, traditional surveillance cameras face limitations when it comes to malicious physical damage or obscuring by offenders. To overcome this limitation, we propose <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m<inline-formula><tex-math notation="LaTeX">$^{2}$</tex-math><alternatives><mml:math><mml:msup><mml:mrow/><mml:mn>2</mml:mn></mml:msup></mml:math><inline-graphic xlink:href="han-ieq1-3445507.gif"/></alternatives></inline-formula> Vision</small>, which is the first millimeter-wave (mmWave)-based video reconstruction system designed to enhance existing video surveillance cameras. <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m<inline-formula><tex-math notation="LaTeX">$^{2}$</tex-math><alternatives><mml:math><mml:msup><mml:mrow/><mml:mn>2</mml:mn></mml:msup></mml:math><inline-graphic xlink:href="han-ieq2-3445507.gif"/></alternatives></inline-formula> Vision</small> utilizes mmWave to sense the profile and motion signature of the target, integrating it with previously acquired visual data about the environment and the target's appearance, thereby facilitating the reconstruction of surveillance video. Specifically, our proposed system incorporates a dual-stage mmWave signal denoising algorithm to efficiently eliminate the noise and multiple-input multiple-output virtual antenna enhanced heatmap generation (MVAE-HG) method to obtain fine-grained mmWave heatmaps responsive to the target's profile and motion information. Moreover, we design the mm2Video generative network that first employs a multi-modal fusion module to fuse the mmWave and pre-acquired visual data, then use a conditional generative adversarial network (cGAN)-based video reconstruction module for surveillance video reconstruction. We conducted comprehensive experiments on <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m<inline-formula><tex-math notation="LaTeX">$^{2}$</tex-math><alternatives><mml:math><mml:msup><mml:mrow/><mml:mn>2</mml:mn></mml:msup></mml:math><inline-graphic xlink:href="han-ieq3-3445507.gif"/></alternatives></inline-formula> Vision</small> using a commercial mmWave radar and four surveillance cameras across various environments, with the participation of seven individuals. Evaluation results show that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m<inline-formula><tex-math notation="LaTeX">$^{2}$</tex-math><alternatives><mml:math><mml:msup><mml:mrow/><mml:mn>2</mml:mn></mml:msup></mml:math><inline-graphic xlink:href="han-ieq4-3445507.gif"/></alternatives></inline-formula> Vision</small> can achieve an average structural similarity index measure (SSIM) of 0.93, demonstrating its effectiveness and potential.