Boosting Rare Scenario Perception in Autonomous Driving: An Adaptive Approach With MoEs and LoRA

Autonomous driving technology has achieved remarkable advancements, offering substantial potential to revolutionize traffic safety and smart mobility. However, when faced with rare scenarios (weather, accident scenes, and lighting), autonomous driving systems can still only play a limited role due to insufficient learning in these rare situations. To address this challenge, we propose a novel approach that leverages low-rank adaptation (LoRA) and Mixture of Experts (MoEs) technologies to enhance the performance of pretrained autonomous driving models in handling rare situations. Specifically, we first use LoRA to fine tune the pretrained model of autonomous driving to focus on capturing knowledge related to rare scenarios and enhance the model’s ability to handle rare situations. Furthermore, we introduce MoEs and propose local, global, and hybrid adaptive solutions to overcome LoRA’s fixed intrinsic rank limitation. These approaches enable adaptive adjustment of LoRA’s rank, and improve the model’s performance from both local and global perspectives. Finally, we design detailed algorithms for different adaptation schemes. Extensive experiments demonstrate that our proposed solutions not only effectively improve the performance of the autonomous driving perception model in rare scenarios but also maintain lower inference latency compared to baseline methods.