Point-MSD: Jointly Mamba Self-Supervised Self-Distilling Point Cloud Representation Learning

Self-supervised tasks, which extract supervisory signals from the data without human labeling, have garnered significant attention. Mask modeling, in particular, has piqued interest. To leverage unlabeled point data in self-supervised learning, we introduce a self-distilling architecture that integrates mask modeling to enhance the model's ability to extract representations and strengthen its high-level semantic feature learning on two versions of a point cloud. Recognizing that point coordinate embeddings can lead to information disclosure distinct from images and text, we incorporate mask embedding into the decoder of an encoder-decoder structure to mitigate this issue and ensure the model's learning is not overly simplistic. These backbones are typically based on CNNs or Transformers, which have limitations in local receptive field and global modeling, respectively. Drawing inspiration from Mamba, which originates from a state space model with global modeling capabilities and linear complexity, we propose a novel approach called Jointly Mamba Self-Supervised Self-Distilling Point Cloud Representation Learning (Point-MSD). Additionally, we introduce a global semantic consistency loss to enrich the overall shape understanding of 3D objects. Extensive experiments demonstrate that our method can learn superior point cloud representations. Notably, Point-MSD achieves a 93.52% accuracy on ModelNet40, outperforming PointMamba by 0.93%.