A compositional dual-hand action recognition method for LLM-driven assembly assistance

Effective assistance in manual assembly requires perceiving the operator’s actions in real time and advising the operator flexibly. Traditional action recognition methods classify a video clip into a single holistic label for one action stream, failing to resolve the parallel dual-hand actions or expose the fine-grained semantics for downstream assistance reasoning. Moreover, conventional assistance systems retrieve predefined content keyed to recognized steps and cannot respond to an operator’s situated needs, whereas large language models (LLMs) can respond flexibly but are too slow and hallucination-prone for the real-time, safety-critical path. This paper connects the two through a compositional action representation that expresses each hand’s action as a structured tuple of action verb , manipulated object , target object , and tool . A compositional dual-hand action recognition model, CoDuAR, is proposed to predict the action tuples of both hands in a single forward pass through a shared encoder and hand-specific compositional decoders. On the HA-ViD benchmark, CoDuAR outperforms state-of-the-art recognizers that require a separate model per hand, while using 23% fewer parameters and roughly doubling the dual-hand inference throughput of the strongest parameter-comparable baseline, VideoMAE V2. Built on the compositional representation, a dual-process assistance system, Assembly Copilot, is developed. A deterministic System 1, driven by CoDuAR, maps the predictions to a symbolic task state, tracking the task progression, recommending next tasks, and detecting operational errors in real time, while an LLM-powered agent, System 2, reasons over the symbolic state to answer the operator’s queries and generate post-session performance reviews, gaining flexibility without placing the language model on the latency-critical path. A case study on a gear-plate assembly validates the functionality of the integrated system. The results indicate that the compositional action representation can serve as an effective interface between real-time perception and language-based assembly assistance within the evaluated scenario.