VIT-DepthSeg: unified panoptic segmentation and depth estimation using vision transformer

In autonomous driving, accurate environment perception and depth estimation are very important for vehicle safety and decision-making, but the existing models often encounter problems such as insufficient perception accuracy, fuzzy depth details and low efficiency. In this paper, a multi-task model combining panoptic segmentation and depth estimation is proposed. Based on the VITarchitecture, segmentation and depth prediction can be obtained in one training. In order to meet a large number of data requirements of Transformer, the ability of feature learning is enhanced with data such as GridMask. Aiming at the ambiguity of depth prediction, a cross-space EMA module is introduced to improve the fine granularity and accuracy of depth estimation. Improve the scale-invariant loss function, optimize performance and enhance robustness. The model provides an efficient and accurate scheme for autonomous driving, which is expected to significantly improve the system perception.