Multimodal Perception and Variational Autoencoder Based Behavior Recognition Method for Cantonese Lifestyle Patterns Abstract

This paper presents a novel multimodal behavior recognition framework tailored for complex lifestyle scenarios, with a focus on Cantonese daily activities. The proposed approach integrates visual, auditory, and inertial sensor data using a feature-level fusion strategy enhanced by a Variational Autoencoder (VAE). The VAE module effectively learns compact, cross-modal latent representations, mitigating modality discrepancy and improving generalization. The fused latent vector is processed by a Bidirectional Long Short-Term Memory (BiLSTM) network, which captures temporal dependencies across sequential behavior segments. To validate the effectiveness of our method, we construct a composite dataset integrating PAMAP2 sensor data, ESC-50 environmental audio samples, and UCF101 video frames, simulating multimodal Cantonese lifestyle behaviors. Experimental results demonstrate that the proposed VAE-BiLSTM model achieves a classification accuracy of 89.7%, outperforming unimodal and traditional fusion baselines. Ablation studies confirm the critical role of VAE-based representation learning and the benefit of multi-source integration. This work contributes a robust and extensible pipeline for human behavior recognition in real-world conditions, combining unsupervised latent modeling and temporal learning. It lays the groundwork for future research in culturally-aware human activity understanding, ubiquitous computing, and smart healthcare applications.