A large language model and augmented reality integrated interface for human-robot collaborative construction waste sorting

Effective sorting is an indispensable component in construction waste material reusing or recycling. By harnessing both robots’ precision and humans’ adaptability, human-robot collaboration (HRC) is widely regarded as a promising approach for construction waste sorting (CWS). However, the lack of an intuitive interface hampers efficient communication between humans and robots. To address this gap, this paper proposes a user-friendly, hands-free interaction method for human-robot collaborative CWS. It does so by integrating a large language model (LLM) into an augmented reality (AR) head-mounted display, enabling human workers to communicate with robots using natural language while visualizing recognition and intentions in an augmented view. A prototype system was developed and tested in several CWS scenarios. It is discovered that the prototype can accurately interpret verbal instructions with 94.03% accuracy and achieve a sorting accuracy of 96.48%. The hybrid use of vision and vocal communication enables near-instant interaction, significantly enhancing the quality and efficiency of CWS. This research advances the field of cleaner materials by reporting a waste recycling and upcycling methodology integrating latest technologies such as HRC, AR, and LLM.