Multimodal Information Fusion based on Event Embeddings and Spatial-Temporal Graph Convolution Networks

The growing diversity of sensor types and information modalities in smart home environments poses significant challenges for effectively fusing heterogeneous data from multiple sources. These challenges stem from variability across sensing channels and substantial differences in data volume, making accurate perception and utilization increasingly difficult. To address this, we proposed UHome Multimodal Fusion Scheme (UHomeMM)—a novel framework that integrates event embeddings with spatial-temporal graph convolutional networks (STGCN) to enhance multimodal fusion in complex environments. UHomeMM fully exploits valuable multi-source heterogeneous information while suppressing irrelevant signals through a cross-channel attention mechanism. Specifically, it encodes sensory data via event embeddings and performs fusion using STGCN to achieve efficient, accurate information integration. Experimental results demonstrate that UHomeMM significantly outperforms traditional methods in fusing heterogeneous data across channels in home environments, effectively reducing noise and enhancing the system's perception accuracy and multimodal feature fusion performance.