Poster: Real-time Keyboard Segmentation and Finger-Press Detection for Keystroke Tracking in Virtual Keyboards

Real-time recognition and localization of keyboard keys combined with real-time hand tracking and detection of finger-press actions are critical functions of vision-based virtual keyboards. These are not trivial tasks because there are numerous types of keyboard layouts and typist behaviors. Furthermore, key detection is especially difficult in uncontrolled real-world scenarios, where users' hands occlude significant portions of the keyboard while typing. Also, the detection finger press-down actions is further complicated in-the-wild by frequently changing camera viewpoints without direct line of sight to the finger pressing a key. In this work, we address the challenge of complete keyboard segmentation and detection of hand-occluded keys by proposing a deep learning approach for realtime finger-press detection, visible-key detection, and occluded-key recovery. Our models were trained and evaluated on Kaggle's Keyboard Key Detection dataset [1] and further empirically tested on the MSU Typing Behavior Database [3]. Our best models achieved 88% finger press-down detection and a key detection performance of 0.91 IOU and 97.8% mAP@75, while running at 60 fps.