VibeGait: Enhancing Structural-Vibration based Gait Recognition using Vision

Structural vibration-based gait recognition has emerged as a promising soft-biometric modality, particularly for privacy-sensitive monitoring and access control. Despite its potential, current research is largely limited to proof-of-concept studies that rely on hand-crafted features, with minimal exploration of deep learning methodologies. This gap reduces the potential for integrating structural vibration-based gait recognition with existing modalities, such as camera-based systems. In this study, we propose a multi-modal gait recognition system that integrates both vision and structural vibration modalities. We address two key challenges: (a) lack of studies exploring outdoor gait recognition using both vision and structural vibration, and (b) absence of a multi-modal training scheme that combines these two modalities. To tackle the first challenge, we curated a dataset comprising five minutes of walking data from ten individuals captured simultaneously by two cameras and a geophone sensor. To address the second challenge, we developed a joint training framework that uses data from both modalities. Our methods achieve an accuracy of 96.03% (±1.12) using structural vibration signals alone, and this improves to 98.27% (±0.06) when both modalities are combined.