Lightweight Vision-language Models for End-to-end Autonomous Driving: An Analysis of Performance With and Without Q-former Training

Recent advances in vision-language models (VLMs) have enabled integrated visual perception and language understanding for autonomous driving. However, the high computational cost of large-scale VLMs limits their practical deployment in real-time systems. In this study, we revisit the common assumption that jointly training the Q-Former improves adaptability in End-to-End autonomous driving. Through empirical analysis, we demonstrate that training the Q-Former can, in some cases, hinder performance by introducing representational bottlenecks or disrupting pre-trained visual semantics. Instead, we show that applying Low-Rank Adaptation (LoRA) to the language model alone achieves better accuracy with significantly fewer trainable parameters. Our findings highlight the importance of careful architectural tuning and suggest that simpler adaptation strategies may be more effective for real-world VLM deployment in autonomous driving.