Social Temperature: Real-Time Social Activity Monitoring Based on Deep Learning Methods

In the study, the social temperature model for measuring social activity of a space first proposed. From observations of crowd (number of people), group atmosphere (positive or negative emotion), and social interaction (conversation), the social temperature model quantizes social activity to the value of social temperature in the range of 0 to 40 degrees Celsius. A real-time camera monitoring system has developed based on deep learning methods to detect motion, human face and identity, face expression and talking voice, and provides an unconstrained social activity monitoring. Deep learning methods of GMM model, YOLOv8n model, Dlib model and Deepface model were adopted in motion detection, face detection, human identification recognition and face expression recognition, respectively. In addition, the OpenAI Whisper-small model was applied in Speech-to-Text conversion to retrieve texts data from audio signals, and Voice Activity Detector run in local devices for distinguishing speech from ambient noise of environments. For real-time computation, a GPU server was built to transfer images and voices from local devices of users via FastAPI. The system has been applied in various authentic contexts including office, school and home scenarios. The contribution of the work provides an objective way for measuring social activity of people and the belonging space in mental aspect that tries to arise people the awareness of their social states with others.