Real-Time Task Detection and Digital Twin Modeling Using Human Posture Estimation in 360-Degree Videos

This paper presents a framework for creating a human-centered digital twin using task detection and human posture estimation (HPE) with Gaussian Splatting from 360-degree videos. Participants performing ladder-related tasks—ascending, working on, and descending a ladder—were recorded using a 360-degree camera. The framework employs HPE to extract key landmark points, and a kinematics-based model detects tasks based on postural transitions observed in the videos. These detected tasks, combined with posture data, enable dynamic visualizations of human movements using Gaussian Splatting. The proposed approach facilitates the creation of a real-time digital twin that visualizes human-environment interactions, capturing the time-series progression of tasks and postures. This method offers scalable real-time monitoring and simulation of human actions, with potential applications in safety training, ergonomics, and human-machine interaction.