Estimation of Finger Angles with PointNet Models Using Point Cloud Data of Forearms

In this paper, we propose a deep learning model that applies PointNet architecture to improve response performance for the occlusion problem beyond the wrist, which was a problem in previous studies. We also evaluate its performance through experiments to estimate finger angles. Positions, joint angles, and accelerations of the fingers are measured and utilized on immersive devices and non-contact interfaces. However, it is difficult to measure them in situations where the wrist is hidden. Therefore, we devise finger angle estimation method that uses point cloud data of forearms as input data. From the results of experiments for 22 participants, the average RMSE was 22.79 and median of R2 was 0.35 when the estimation was performed using a trained model. It suggests that the proposed model can estimate finger angles from the three-dimensional shape of forearms. Moreover, the time required for processing one estimation was 3.798 ms, which indicates that the response performance was good enough.