Turning Gesture Sequences into Gesture Composition by Macro-Instructions: Definition, Framework, and Application in Smart Home EICS016

2D-3D gesture interaction most often associates a recognized gesture with a single command, such as “Swipe right” to navigate to the next screen. When gestures are performed one after another in a gesture sequence, each gesture still needs to be associated with one separate command at a time. Increasing the number of such gestures to realize more commands is not favorable to the end user, who memorizes and uses only a limited number of gestures. Instead of relying on gesture sequences, we can expand the range of executable commands to perform more complex tasks by gesture composition. To this end, we define an instruction as an action applied to a device in an environment, together with its valued parameters, such as “Increase the brightness of the bedroom lamp to 10“. Elementary gestures, preferably congruent and hierarchical, are composed into a compound gesture. The instructions can be combined into macro-instructions to summarize gesture composition. To put these definitions in practice, we developed Emeriti , a framework for the recognition of elementary and compound gestures in both 2D (uni- or multistroke gestures recognized on an interactive surface) and 3D (point trajectories in space) by composition. We demonstrate the effectiveness and usefulness of turning gesture sequences into gesture composition using (macro-)instructions in an application simulating a smart home in which every device in every room can be controlled by compound gestures with parameters. This makes it possible to multiply the number of possible commands for more complex tasks without considerably increasing the number of gestures.