Generative Sensing: Pre-training LiDAR with Masked Autoencoders for Ultra-Frugal Perception

We propose a disruptively frugal generative sensing approach for LiDAR that generates, rather than senses, parts of the environment that are either predictable based on extensive training or have limited impact on overall prediction accuracy. Our generative pre-training strategy for this purpose, radially masked autoencoding (R-MAE), also allows focusing on radial segments of the data, which captures spatial relationships and distances between objects more effectively than conventional procedures. As a result, the proposed methodology not only reduces sensing energy but also improves prediction accuracy. Our evaluations on Waymo and nuScenes show that our approach achieves over a 5% average precision improvement in detection tasks across datasets and over a 4% accuracy improvement when transferring domains from Waymo and nuScenes to KITTI. Our method achieves up to 3.17% and 2.31% improvements in mean average precision (mAP) and NDS, respectively, in nuScenes. Even with 90% radial masking, it surpasses baseline models by up to 5.59% in mAP and mean average precision with heading (mAPH) across all object classes in the Waymo dataset. Resultantly, R-MAE reduces the sensing energy by ∼9.1× than the conventional LiDAR processing despite computational overheads. Codes are available: https://github.com/sinatayebati/Radial_MAE.