From Depth to Flow - Learning Uncertainty Aware Stereo Scene Flow from Partial Synthetic Labels

Estimating both the 3D structure and 3D dynamics of a scene is an important task for autonomous vehicles and mobility systems. Deep learning based methods have shown strong performance on this problem, but are only reliable in their training domain and often do not reason about uncertainty. Training data that contains information about the detailed dynamics of a scene is often difficult and costly to acquire, thus limiting the applicability of such methods. To address these issues, we propose an uncertainty-aware multi-task neural network that jointly estimates disparity, optical flow and 3D scene flow from stereo image pairs in a single model with shared weights for all tasks. Furthermore, we investigate how labels of individual subtasks, e.g. disparity, can be used in combination with self-supervised losses to improve the performance of other subtasks, such as optical flow. We show that a unified model such as ours can leverage these supervision types synergistically, to transfer knowledge even from simpler tasks to more challenging ones. Additionally, we propose the first scene flow approach that estimates uncertainties as variances and multivariate covariance matrices from the cost volume of each respective task and propagates them analytically to the pixel-wise output without any further learned regression. We evaluate the domain adaptability and pixel-wise uncertainty estimations of our model on both synthetic and real datasets, including the KITTI scene flow benchmark, on which our model outperforms prior self-supervised and semi-supervised methods, while estimating representative uncertainties for all tasks.