Automated Data Labelling for Pedestrian Crossing and Not-Crossing Actions Using 3D LiDAR and RGB Data

The existence of large-scale datasets for various autonomous driving tasks has created an increasing need for more automated annotation processes. Especially for safety critical tasks related to vehicle-pedestrian interaction, detailed and time-consuming human-made annotation is required, in order to assure accurate perception throughout any type of operating environment and for challenging conditions. In this paper, we present an automated method for the annotation of actions of humans crossing or not crossing the road. Firstly, we utilize a highly-accurate 3D multi-object tracking pipeline that combines RGB images and LiDAR data to extract the velocity and direction of movement of each pedestrian in the surrounding environment. A drivable area extraction neural network is then utilized to segment the traversable area around the vehicle. The correlation between the two above-mentioned components in the 3D space provides an accurate indication, regarding the pedestrian crossing or not-crossing the road ahead of the vehicle. Our method is validated using a custom-made multimodal dataset with an autonomous vehicle in various scenarios of a semi-structured area. The auto-generated annotations are compared directly with the human-made labels of multiple annotators and showcase the effectiveness of our method to provide an accurate indication about the human crossing the road action.