A novel lightweight skeletal temporal model for real-time, computationally efficient recognition of occupant thermal adaptation behavior

Abstract Optimizing building energy systems based on real-time occupant behavior and feedback can lead to improved energy efficiency and enhanced thermal comfort in buildings. Traditional thermal comfort surveys do not provide real-time insights, while conventional sensors, such as thermal sensors, are limited in their ability to capture continuous, detailed occupancy data. Meanwhile, deep learning and computer vision have emerged as promising approaches for real-time occupancy behavior detection, but existing artificial intelligence (AI) models suffer from low frame rates and high computational demands, which can lead to increased energy consumption for processing, potentially offsetting the energy savings achieved through occupant-responsive control. Thus, this study developed a novel occupant thermal adaptation behavior recognition model that balances accuracy, real-time performance and computational resource usage to enable effective operation indoors. Using a multi-camera setup with Raspberry Pi 3B+, a custom dataset comprising 400 video samples was collected from four different angles. The dataset captures four distinct human activities: dressing, undressing, sitting, and standing. Compared to SlowFast (SF) and Spatial Temporal Graph Convolutional Networks (ST-GCN), which are widely used deep learning architectures for action recognition, the proposed novel lightweight skeletal temporal model achieved good accuracy (0.975 accuracy) on the Kungliga Tekniska Högskolan (KTH) dataset while significantly outperforming them in detection speed and resource efficiency. It reached 31.38 FPS by running on the graphics processing unit (GPU)—over three times faster than ST-GCN with OpenPose and more than twelve times faster than SF with You Only Look Once Version X (YOLOX)—while maintaining low central processing unit (CPU) and GPU usage at 13.71% and 33.05%, respectively. By running it on the CPU, it achieved 25.3 FPS with 56.10% CPU usage, proving its practicality for platforms without GPU support. When evaluated on the custom dataset, we introduced a double long short-term memory (LSTM) with an attention mechanism to better handle the increased action complexity, preserving a high accuracy of 0.963. Although the frame rate experienced a slight reduction compared to the results on the KTH dataset—dropping from 31.38 to 30.95 FPS on GPU and from 25.3 to 18.98 FPS on CPU—the model exhibited lower CPU and GPU usage, highlighting its potential for energy-efficient deployment in smart building applications. The model was further deployed on an NVIDIA Jetson Orin Nano, enabling stable long-term operation and supporting simultaneous multi-person recognition. Overall, this study presents a practical, AI-driven solution for occupant thermal adaptation behavior recognition, effectively balancing accuracy, real-time performance, and computational efficiency—making it well suited for energy-saving applications in buildings that adapt dynamically to occupant behavior.