TrustHeartbeat: A Trustworthy Multimodal Interactive System with Heartbeat Recognition and User Identification

The growing popularity of cardiac sensors has led to an increasing demand for smarter interactive systems in disease diagnosis. Various cardiac sensors have emerged as alternatives for heartbeat monitoring and cardiac disease diagnosis, but existing single-modal systems often exhibit biases in real-world applications. In this paper, we propose TrustHeartbeat, a multimodal interactive system that fuses multiple cardiac signals to recognize and identify heartbeat patterns, offering more robust and convenient interaction. TrustHeartbeat is built on a Multimodal Masked Autoencoder (MultiMAE), capable of reconstructing multi-modal heartbeat signals even when some modalities are unavailable. We further enhance the MultiMAE model by replacing the decoder with a global classifier for classification tasks. TrustHeartbeat fuses embeddings from multiple modalities for downstream tasks, such as heartbeat classification and user identification. A prototype of TrustHeartbeat was implemented and evaluated, with extensive experiments demonstrating its effectiveness. The system achieved F1-scores of 85% for user identification and 92% for heartbeat recognition, maintaining robustness across different experimental settings.