Facial Expression Recognition with DToF Sensing

Facial Expression Recognition (FER) is crucial for understanding human emotions, with applications spanning from mental health assessment to marketing recommendation systems. However, existing camera-based methods raise privacy concerns, while RF-based approaches suffer from limited environmental generalizability and high cost. In this work, we propose ToFace, a FER system leveraging a low-cost (4.8$) Direct Time-of-Flight (DToF) sensor that has been available on commodity smartphones. This sensor provides an extremely low-resolution 8 × 8 depth map and a clear Field of View (FoV), significantly mitigating privacy concerns while avoiding the impact of ambient objects. Despite the benefits, the low-resolution depth map introduces significant challenges for precise expression recognition due to limited facial structure information. We first develop a physical model to extract additional spatial information from the intermediate sensor output, i.e., the transient histograms. We then propose a physics-integrated neural network to reconstruct a facial structure map comprising both depth and orientation for accurate expression recognition. We conduct real-world experiments with 12 users and compare our model with several baselines. The results demonstrate that ToFace achieves the highest recognition accuracy of 75%.