Unified Self-supervised Representation Learning for Multi-modal and Single-modal 3D Perception

Constructing large-scale labeled datasets for multi-modal perception in autonomous driving is challenging, driving the need for self-supervised pretraining. However, existing methods typically adopt separate pretraining strategies for each modality. Inspired by Neural Radiance Field (NeRF), which unifies multi-modal optimization, we propose NeRF-Supervised Masked Autoencoder (NS-MAE), a unified pretraining strategy for LiDAR-Camera 3D perception. NS-MAE encodes both appearance and geometry, enabling efficient masked reconstruction across modalities. Specifically, it extracts embeddings from corrupted LiDAR point clouds and images, conditioned on view direction and location. These embeddings are then rendered into multi-modal feature maps from two key perspectives for autonomous driving: perspective view and bird’s-eye view (BEV). The original uncorrupted data serve as reconstruction targets, facilitating self-supervised learning. Extensive experiments demonstrate NS-MAE’s superior transferability across various 3D perception tasks. Notably, NS-MAE outperforms prior SOTA pretraining methods which rely on separate modality-specific strategies, particularly in BEV map segmentation under label-efficient fine-tuning. Our code is publicly available at https://github.com/Xiaohao-Xu/Unified-Pretrain-AD/.