mmHolmes: Amodal Millimeter-wave Sensing by Understanding Human Kinetics

Though mmWave sensing shows huge potential for recognizing human activities in an advantageous contact-less, privacy-protection manner, its performance in daily life household scenarios has not been fully understood yet. To understand the performance of mmWave sensing in the wild, we conduct a measurement campaign lasting 1-month duration in 4 daily living environments. It reveals that existing mmWave sensing methods fail to recognize partially-active activities that take a dominant ratio of 80% of all daily activities. The reason is that they neglect human kinetics with multiple degrees of freedom, but mistakenly treat the movement of partially-observable body parts as that of the whole body. To resolve the problem, we propose mmHolmes, a fundamentally different mmWave sensing methodology with amodal sensing ability, i.e., comprehending complete human structure from partial visibility. To endow mmHolmes the amodal sensing ability, we design a self-supervised pre-training framework that can effectively learn human kinetics, i.e., the spatial and mechanical relationships among different parts of the human body, from massive unlabeled activities of daily living, which finally leads to accurate sensing of human activities in practice. We prototype, deploy, and evaluate mmHolmes extensively, which demonstrates ~20% higher accuracy than the state-of-the-art, and also remarkable robustness in terms of distinguishing unseen and easily confused activities across various conditions.