Monocular underwater vision pipeline for 6DoF annotations with inpainting-based image augmentation

The acquisition of high-fidelity, annotated data for training perception and manipulation tasks poses significant challenges. This process typically demands customized setups, tightly controlled environments, and specialized sensing equipment that are unavailable in underwater settings. Marker-based methods offer a simpler alternative by tracking the six degrees of freedom poses of objects using a monocular camera. However, attaching markers to objects alters their original form and appearance, while placing markers in the environment modifies the backdrop and limits the flexibility and portability of such methods. In this work, we present a pipeline capturing underwater scenes using a pose plate with fixated featureless objects of varying scales. The pose plate is equipped with ArUco markers, which track the 6D camera pose and enable the pipeline to render pixel-wise depth and object masks. Custom camera mappings ensure precise alignment between rendered masks and sensor images. To prevent machine learning models from relying on the markers as cues rather than building robust object representations, our pipeline employs object aware inpainting as augmentation method, replacing the pose plate with a realistic background. The pipeline was validated by training semantic segmentation models on a custom dataset consisting of scenes in different underwater environments. Our experiments demonstrate that incorporating augmented data into the training process yields improved model performance, outperforming models trained solely on images with visible markers. This finding suggests that our proposed techniques have the potential to mitigate the domain gap between marker-based ground truth and real-world data.