MtpNet: Multi-Task Panoptic Driving Perception Network

Panoramic driving systems are crucial for autonomous driving but face challenges in real-time performance and reliability. This paper proposes an end-to-end, multi-tasking MtpNet that reduces latency and enhances detection accuracy. The convolution was upgraded using the Efficient Layer Aggregation Network, and precise multi-task loss functions and more effective training strategies were devised. Our results demonstrate improved performance in small object detection, partial occlusion handling, and drivable area segmentation. The recall of the traffic object detection is 1.3% higher than that of the state-of-the-art model, reaching 94.1%, the mAP<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">50</sup> is 6.4% higher, reaching 89.8%, and the mIoU of the drivable area segmentation is 2.7% higher, reaching 95.9%. Additionally, the accuracy of lane detection reached 88.7%. The visual comparison using three datasets TuSimple, CityScapes, and CULane demonstrates that MtpNet has good detection segmentation and strong robustness under various conditions. Codes are available at https://github.com/ErLinErYi/mtpnet