Using Scene‐Flow to Improve Predictions of Road Users in Motion With Respect to an Ego‐Vehicle

ABSTRACT We addressed the challenge of accurately determining the motion status of vehicles neighbouring an ego‐vehicle, across various driving scenarios. The aim was to enhance the prediction accuracy in identifying moving vehicles through the integration of scene‐flow analysis into tracking. The research was motivated by the importance, in autonomous driving, of analysing the state exclusively of moving vehicles. We implemented a novel, synergistic, vision‐based, and offline approach, named MoVe, combining spatial analysis of predicted scene‐flows and temporal tracking, from sensor‐fused input data. Regions of moving vehicles (post background refinement) were obtained via instance segmentation, and each instance mapped to the corresponding (original) scene flows. Our method achieved an 1 score of 0.953 and accuracy of 0.959 for binary motion classification (stationary vs. moving). The proposed fusion segmentation model produced an mIoU of 82.29% for cars, outperforming YOLOv7 which relies solely on visual features. Notably, we observed a complementary dynamic between scene‐flow analysis and tracking. Scene‐flow analysis was generally effective in identifying fast moving vehicles, even under occlusions or truncations caused by other vehicles or infrastructure elements, while tracking usually excelled in identifying comparatively slow moving vehicles. Thus, the study demonstrated the viability of our proposed architecture to improve the detection of moving vehicles around an ego‐vehicle. The outcomes further suggested the potential of our work to be utilised for training future deep learning models based on machine vision and attention, such as object‐centric learning, which paves the way for enhancing perception, intent estimation, control strategies, and safety in autonomous driving.