Mutli-modal Human Pose Estimation

3D Human Pose Estimation (HPE) in autonomous vehicles differs significantly from other domains such as gaming or VR. In the context of autonomous vehicles, 3D HPE involves understanding of human poses and actions in real world “in the wild” scenarios within the surroundings of the vehicle. This paper particularly suggests a solution for this issue in the AV context. The image is first passed through Mask R-CNN to first segment pedestrians on RGB photos, and then this information is combined with LiDAR, along with an auxiliary segmentation branch for 3D HPE to potentially improve the accuracy, as Mask R-CNN can segment the pedestrians in the images and provide better spatial information about their location in the scene.