Flexible multitask deep learning for lane-aware panoptic segmentation in autonomous driving

Video panoptic segmentation (VPS) is a computer vision task that provides rich spatiotemporal information about the surroundings of an autonomous vehicle at the pixel level. Lane semantics, such as lane markings, lane classes, etc ., help in complex driving decisions. The state-of-the-art VPS networks that rely on VPS datasets that segment all lanes into a single road class, and hence cannot infer necessary lane semantics. In addition, the joint learning of heterogeneous subtasks like tracking and segmentation of VPS has drawbacks like single-task performance degradation, laborious fine annotations for the dataset, inflexibility in incorporating new lane semantics, and manual tuning of loss weight optimization. The proposed work presents a decoupled approach for the VPS task and includes lane semantics into VPS. This work segments the lanes into three classes, namely ego, left, and right lanes, using a new dataset termed the left-ego-right (LER) dataset. The primary challenge was discriminating the relative direction of the ego-vehicle concerning the road since it is a bland feature. This was addressed by introducing novel semantic weights during training of the semantic segmentation network for lane segmentations. The results demonstrate better discrimination of different lanes compared to networks that do not involve semantic weights, with 2.42% improvement in intersection over union (IoU) and 5.22% improvement in mean average precision. Improvements demonstrated on other popular metrics for semantic segmentation presented in this article assure direction-learning enhancement of semantic segmentation tasks in general.