Semantic Highlighting: Visual Guidance in Multimodal Gaze-Keyboard-Based Interaction

Modern video-based eye trackers have reached a level of accuracy sufficient for gaze-based human-computer interaction. In many but not all applications, accuracy even allows for replacing a computer mouse with gaze-based pointing. The Multi-Modal Interaction Concept for Efficient Input (M<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>ice) tackles accuracy problems with an on-demand hybrid linear-fisheye magnifier. This paper explores how semantic analysis of screen content can be used as part of an alternative solution to the accuracy problem. This solution integrates machine-learning-based near real-time image analysis into the M<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>ice interaction concept and adds semantic highlighting to screen elements near the user's interaction point of gaze. Semantic highlighting identifies clickable screen elements and adds a number as well as a bounding box to each of these elements. This allows users to select elements with a keystroke. It thus combines visual magnification with visual semantic enrichment. The image analysis is based on a machine learning model specifically trained for the respective use case. A user study <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\mathrm{n}=20)$</tex> showed no significant speed difference with the use of semantic highlighting but a 94.3 % decrease in interaction errors (misclicks) with the use of semantic highlighting compared to the traditional selection and magnification approaches.