From detection to intervention: An end-to-end system for recognizing the “signal for help” gesture in real-time

• Expansion of the “Signal for Help” gesture dataset. • Two-step pipeline system for real-time “Signal for Help” gesture detection. • Mobile app delivers instant video alerts and precise location to security personnel. • Validated system ensures reliable performance across edge and central platforms. The “Signal for Help” is a simple hand gesture, internationally recognized, that enables individuals experiencing domestic violence to discreetly signal their need for help without alerting their aggressors. Developed during the COVID-19 pandemic to address the growing isolation of victims, it serves as a powerful tool to facilitate silent communication in dangerous situations. Despite its potential, its effectiveness has been impeded by limited public awareness, the risk of misinterpretation, and the lack of reliable automated detection systems. To address these challenges, this paper introduces a framework consisting of two interconnected components: a real-time detection system of the “Signal for Help” gesture using a machine learning-based recognition system and a custom mobile application that receives notifications from the detection system and alerts security personnel in real-time. During the development process, we faced several challenges, including detecting the gesture in crowded environments and keeping the computational load low to ensure the system could run efficiently on edge devices. We overcame these challenges by designing a system that combines hand tracking and feature extraction, using tools such as MediaPipe and DeepSORT, followed by a final classification step. After testing various classifiers, Random Forest achieved the best results, reaching an accuracy of 94% with a very low rate of false positives. The system was carefully optimized to minimize computational cost while maintaining real-time performance. In fact, as shown by the tests conducted on Apple M3, NVIDIA Jetson Orin Nano, and NVIDIA Jetson AGX Orin, the system achieved inference times of 0.067 seconds, 0.471 seconds, and 0.343 seconds respectively. These outcomes demonstrate the system's possibility for deployment in smart city environments, supporting both urban and non-urban areas. When a gesture is detected, the system immediately notifies the mobile application, which provides instant alerts, geolocation data, and a short video clip of the incident, enabling a rapid and informed response. Additionally, the app includes advanced features such as detailed notification history, real-time operator status monitoring, and an integrated team coordination chat, which optimize operations, enhance collaboration among security staff, and ensure timely and effective interventions in emergency situations. This research marks a step forward in real-time gesture recognition and intervention, setting a new benchmark for automated safety systems aimed at preventing domestic violence and other emergencies. By increasing awareness and ensuring a rapid response to the “Signal for Help” gesture, the system empowers individuals in distress and contributes to safeguarding those at risk.