Generalizable Skill Learning for Construction Robots with Crowdsourced Natural Language Instructions, Composable Skills Standardization, and a Large Language Model

The quasi-repetitive nature of construction work and the resulting lack of generalizability in programming construction robots present persistent challenges to the broad adoption of robots in the construction industry. Robots cannot achieve generalist capabilities because skills learnt from one domain cannot readily transfer to another work domain or be directly used to perform a different set of tasks. Human workers have to arduously reprogram their scene-understanding, path-planning, and manipulation components to enable robots to perform alternate work tasks. The methods presented in this paper resolve a significant proportion of such reprogramming workload by proposing a generalizable learning architecture that directly teaches robots versatile task-performance skills through crowdsourced online natural language instructions. A large language model (LLM), a standardized and modularized hierarchical modeling approach, and a building information modeling-robot semantic data processing pipeline are developed to address the problem of multitask skill transfer. The proposed skill standardization scheme and LLM-based hierarchical skill learning framework are tested with a long-horizon drywall installation experiment using a full-scale industrial robotic manipulator. The resulting robot task learning scheme achieves multitask reprogramming with minimal effort and high quality.