Cross-attention spatial–temporal convolutional neural network for energy expenditure estimation on the basis of physical fitness characteristics

Energy expenditure estimation can be used to measure the exercise load and physical condition of different individuals, such as soldiers, athletes, firemen, etc., during their training and work. Energy expenditure estimation methods based on computer vision have rapidly developed in recent years. Compared with sensor-based methods, such methods are capable of monitoring several target persons at the same time and the subjects do not need to wear different sensor devices that hamper their movement. In this paper, we propose a cross-attention spatial–temporal convolutional neural network to predict the energy expenditure of people under different exercise intensities. The model explores the relationship between changes in the human skeleton and energy expenditure intensity. In addition, a cross-attention correction module is used to reduce the negative effects of individual physical fitness characteristics during energy expenditure estimation. The experimental results show that our proposed method achieves high accuracy for energy expenditure estimation and performs better than existing computer vision-based energy expenditure estimation methods do. The proposed method can be widely used in various physical activity scenarios to measure energy expenditure, increasing the convenience of usage.