Navigating Interfaces: Gaze and Gesture as Multimodal Inputs

The project Multimodal Maestro believes in a groundbreaking system of human-computer interaction (HCI) for the benefit of common users and people with disabilities. It provides two visual input methods which includes eye movement and hand gesture, the choice is made by the users based on their convenience and accessibility needs. For general users, Multimodal Maestro opens up a new realm of interaction allowing for a more intuitive and immersive connection with technology. To achieve this we have used the advanced embedded machine learning algorithms such as CNN, Douglas-Peucker, EAR(Eye Aspect Ratio)- based LDA(Landmark Detection Algorithm) and LSTM(Long Short Term Memory) which decodes gestures and very subtle eye movements, including blinks and shifts in gaze into commands including cursor movement, click, and screen element selection [3]–[28]. In tandem, hand gestures provide a natural and expressive alternative way to communicate with technology, thus enhancing the user experience, offering seamless handsfree control. For a person with disabilities, Multimodal Maestro is a game-changer, this will provide a very accessible tool for people with limited mobility to have a way to interact with technology without relying on any traditional input devices[10]. Gaze-based control gives a powerful and inclusive option for mobility-impaired users, while hand gestures provide an additional channel of communication with maximum freedom and independence offering flexibility and personalization, from these two complementary input methods, which is also a suite of adaptive, personalized, and accessible interface. For the first time, this system aims to make technology accessible to all users irrespective of their capabilities and preferences, directing a new challenging path where users can interact with computers in a way that is more natural, empowering and efficient.