HM-SYNC: A Multimodal Dataset of Human Interactions With Advanced Manufacturing Machinery

Abstract A deep understanding of manufacturing processes is essential for advancing manufacturing-oriented design and engineering complex systems. As advanced manufacturing technologies evolve, systems have grown more complex, and human interaction has become a vital component of both operation and design. This shift introduces new challenges as human roles within these systems extend beyond traditional boundaries and are not yet fully understood in current design processes. Characterizing human interactions within manufacturing systems is therefore critical to supporting further advancements. Additionally, human behavior plays a significant role in many engineered systems beyond manufacturing, underscoring the value of developing methodologies to better analyze human behavior and interactions within complex environments. These methodologies can broadly support and enhance diverse aspects of engineering design. This study presents HM-SYNC, which is a comprehensive dataset of human interactions with advanced manufacturing machinery, specifically a wire-arc additive manufacturing machine. Depth images and 3D skeleton joints are collected over 6 months using privacy-preserving pose tracking with depth cameras. HM-SYNC includes thousands of interactions across various contexts, goals, and users, providing valuable insights into patterns of human–machine interaction. By capturing a diverse range of interactions in natural settings, this dataset supports advancements in human-centered manufacturing design and facilitates the development of more effective manufacturing systems. This dataset can enhance models and digital twins of manufacturing systems, help operators optimize machinery use and efficiency, and guide designers in refining machine and system design, to name just a few applications.