Graph-Based Human–Object Interaction Detection for Automated Worker Accountability Monitoring on Construction Sites

Effectively monitoring worker accountability in construction project controls requires addressing the complex, dynamic interactions between workers and objects. However, existing human–object interaction (HOI) approaches are typically limited to single actions and single-scale object sizes, lack spatial context, and overlook how large slab structures require distinguishing between local interaction areas (local objects) and broader object zones (global objects). To overcome these challenges, this study proposes a comprehensive HOI detection framework that leverages convolutional neural networks and graph attention networks to generate a robust worker accountability monitoring (WAM) system. The research identifies various worker actions, such as tying, transporting, and pouring, as key performance indicators, while also capturing noninteractive behaviors to provide a comprehensive view of on-site activity. Objects are grouped into three categories—concrete, formwork, and steel rebar—and analyzed across three object sizes: big, medium, and small. Additionally, the bounding box design of the objects incorporates both local and global scenarios, which consider the area of the associated objects. The system achieves notable performance, with a mean average precision (mAP) of 0.830 for object detection and mAP scores of 0.553 and 0.502 for HOI tasks involving human interactions in local and global contexts, respectively. These results demonstrate accurate detection and action recognition across diverse scenarios. The WAM system generates essential data: worker, object, and action information supporting holistic project management systems, including productivity monitoring, quality control, and safety inspections. By enabling managers to better understand worker interactions and crew activities on construction sites, this approach provides actionable insights to enhance future performance.