Semantic Highlighting: Using Semantic Context Detection for Visual Guidance in Multimodal Gaze-Keyboard Interaction

Modern video-based eye trackers have reached a level of accuracy sufficient for gaze-based human-computer interaction. In many but not all applications, accuracy even allows for replacing a computer mouse with gaze-based pointing. The Multi-Modal Interaction Concept for Efficient Input (M[Formula: see text]ice) tackles accuracy problems with an on-demand hybrid linear-fisheye magnifier. This paper explores how semantic analysis of screen content can be used as part of an alternative solution to the accuracy problem. This solution integrates machine-learning-based near real-time image analysis into the M[Formula: see text]ice interaction concept and adds semantic highlighting to screen elements near the user’s interaction point of gaze. Semantic highlighting identifies clickable screen elements and adds a number as well as a bounding box to each of these elements. This allows users to select elements with a keystroke. It thus combines visual magnification with visual semantic enrichment. The image analysis is based on a machine learning model specifically trained for the respective use case. A user study (n[Formula: see text]=[Formula: see text]20) showed no significant speed difference with the use of semantic highlighting but a 94.3[Formula: see text]% decrease in interaction errors (misclicks) with the use of semantic highlighting compared to the traditional selection and magnification approaches.