A Novelty Real-Time Gesture Recognition Model for Air-Hand Piano Playing Using Mediapipe

This paper proposes an artificial intelligence model designed to identify the positions of the fingers on the hands, assigning them corresponding musical notes. The creation of this model involved the development of a custom dataset comprising various hand configurations, each specifying how a person mimics pressing a piano key. To expand the volume of data and improve its variety, data augmentation was used, a technique that helps the model perform more efficiently under various conditions of use. The data processing included normalization of the coordinates relative to the wrist, so that aspects such as the size, color or shape of the hands do not influence the performance of the model, and the actual position of the hands is independent of the distance from the camera. Another innovative aspect of the development was the application of contrastive learning, whereby the model distinguishes between correct and incorrect hand positions, forming pairs of similar data and minimizing the distances between their coordinates. The results show that the model achieved a high performance for hand gesture classification with an accuracy of 98.89%.