An Online Hard Object Mining and Synthesis and Selection Method for Lidar-based 3D Object Detection

Models and data are two critical factors affecting the performance of Lidar-based 3D object detectors. So far, numerous studies have been made to study the model network structure, and many classic 3D object detectors have been proposed and verified i n practice. However, t he research on improving detectors' performance through data synthesis and augmentation is not sufficient. Detectors that perform w ell on offline d atasets o ften encountering a degradation i n t he complexities and variability real-world, especially when faced with scenes that were rarely present in the training set. Therefore, this paper proposes an online technique for mining, synthesis and selecting hard objects, which can help detectors find hard objects at different stages during training and synthesize scenes with those hard objects by domain randomization and then select the synthetic scenes with diversity and low redundancy. By doing this, the model is guided by synthetic data to train towards enhancing the detection capability of difficult objects. The method includes three modules: online hard objects mining, scenes with hard objects synthesis and data selecting. First, during training process, hard objects that are highly ambiguous to the current model will be identified due to their contribution to the training loss. Subsequently, diversity scenes with hard objects will be generated by randomly combining multiple objects domains. Then, the synthesized data will be sampled to reduce redundancy and mitigate domain shifts. Finally, selected data will be mixed with the original data for next training iteration. Our experiments conducted on the nuScenes dataset demonstrate the effectiveness of the proposed method: different lidar-based 3D object detectors achieved improvements with an average increase of 3.6% in mAP and 2.3% in NDS.