M2FU: Multi-Modal Fusion for Urban Autonomous Driving

Existing sensor fusion approaches are poor, this paper proposes a novel end-to-end autonomous driving model called M2FU. This model solves the problem of past end-to-end models that have many collisions, are heavily blocked, and are unable to effectively detect traffic lights. The perception module of the M2FU improves on the transformer-based detection model by using a dual-attention mechanism for its intermediate feature fusion and a recurrent neural network with gated recurrent units for path planning. In addition, it includes two controllers for managing the vehicle’s speed and steering angle. The model was trained using an imitation learning approach and subsequently evaluated on the CARLA autonomous driving benchmark. The results demonstrate that the end-to-end model based on the dual attention transformer outperforms traditional classifier-based models in avoiding dynamic obstacles. We validate the effectiveness of our method through experiments in urban environments with complex scenes, utilizing the CARLA urban driving simulator. Our driving scores improved by 17% compared to geometry-based fusion.