Temporal- and Appearance-Guided Object Detection in Construction Machines Considering Out-of-Distribution Data

The automation in the construction machine field requires a robust understanding of their surroundings and should be able to localize and classify surrounding objects robustly. State-of-the-art object detection algorithms are usually deep learning–based approaches that take red–green–blue (RGB) images as the only input. However, recent findings highlighted the limitation that these deep learning–based approaches may not perform robustly due to the bias introduced by the training set, which is also a common problem in existing construction machine data sets. As a result, the object detection performance on out-of-distribution (OOD) data is significantly worse than on the training set. This may cause severe accidents and unexpected economic losses on smart construction sites. To address this issue, this study proposes a novel object detection algorithm, called “Temporal- and Appearance-Guided Object Detection” (TAG), which optimally extracts information from temporal information (optical flow) and appearance information (RGB) to improve object detection accuracy and robustness despite OOD data. To evaluate the performance on various OOD data sets, a custom construction machine data set generation system is created that enables nonoverlapping training and testing distributions. Tests with the simulated data set and the real-world data set are performed considering the diversity of the working conditions and the challenge of the OOD data. Compared with existing typical alternative solutions, the results show strong empirical evidence that the proposed construction machine object detection algorithm significantly increases the robustness and generalization capability in dynamic cases without compromising performance in static cases.