WowSense: A High-Accuracy Real-Time Grip-State Sensing on Commodity Smartphones

Smart-devices' grip-state has shown great potential to enable various intelligent applications, including virtual keyboard and automatic UI adaption. However, smartphones nowadays often lack the capability to detect the grip-state, resulting in a poor experience of human-computer interaction. To implement an effective and efficient grip-state detection, we need to tackle a number of technical challenges such as effective features extraction and fusion from multimodal data, nonalignment of different modal data, and limited labeled data availability. In this work, to address these challenges, we design a two-stage grip-state detecting system, named WowSense, for high-accuracy, real-time detection of phone's grip-states using IMU (Inertial Measurement Unit) and CS (Capacitivc Screen) data. Our system WowSense consists of the multimodal alignment stage and the grip-state classification stage. In the first stage, we employ a novel augmentation method to capture subtle features from IMU and CS data. Additionally, we utilize contrastive learning to extract consistent information across these two modalities using a large amount of unlabeled data. In the second stage, we design an attention-based classifier to capture complementary information using only a small amount of labeled data. We implement our system in OpenHarmony and our extensive experimental results demonstrate the superiority of our system, which achieves 95 % accuracy with only 40 % of the data labeled on a self-collected dataset and 92.5 % accuracy with a latency of only around 10ms when running in real-time on a phone,