Emotion-Aware Systems for Hazardous Operations: Integrating Mobile Sensors with LLM

This study proposes a mobile device sensor-based emotion sensing system for hazard-involved operational action environments. In this study, the system utilizes accelerometers and gyroscopes on mobile devices for actor gait data acquisition. The obtained gait data is classified using an artificial intelligence algorith to classify the emotional state of the operator. At the same time, the system combines the gestures recognized by the sensors to identify critical operations, and combines the generative large language model (LLM) to provide personalized reminders and safety recommendations in high-risk operations. Through data analysis, the prediction accuracy of the regression model reaches 68% in anxiety, anger or relaxation, real-time safety reminders based on emotional state. Edge computing-based processing analyzes sensor data in real-time and combines emotions and actions to generate adaptive reminder content. This approach demonstrates the potential to develop low-cost, real-time methods for monitoring an individual’s mental health using LLMs.