Correlation-guided Masked Autoencoder with Multimodal Contrastive Interaction on Point Clouds

Self-supervised learning has shown remarkable effectiveness in 3D point cloud understanding. Existing masked autoencoders or contrastive learning paradigms can acquire robust feature representations from unlabeled data. Specifically, masked autoencoders extract features of local patches and directly map them to latent global vectors, suffering from insufficient semantic extraction and latent interaction. Contrastive paradigms capture global correspondence via restricted constraints and are limited by the absence of local detail modeling. This prompts us to integrate the synergistic local and global advantages of two effective components and extend them further for multimodal dependencies. In this article, we propose a unified correlation-guided masked autoencoder with multimodal contrastive interaction (CorMAC) learning for self-supervised point cloud analysis. We first design the spherical adaptive embedding backbone to learn local underlying semantics and improve the masked mechanism for patch autoencoding and reconstruction. Then, we expand multimodal contrastive correspondence and constraints to leverage the potential alignments across point cloud and auxiliary image modalities. In addition, we devise adaptable loss functions to jointly optimize masked recover and contrast errors, aiming to enhance latent feature learning. Extensive experiments show that our method achieves superior performance than other self-supervised ones on various datasets and exhibits better generalization capability across diverse downstream tasks.