LLM-DNN Fusion for Data-Efficient Trajectory Generation in Autonomous Driving: A Hybrid Approach with MPC Optimization

Autonomous vehicle (AV) trajectory planning in dense urban environments requires models that generalize to rare or previously unseen scenario types. Deep neural networks (DNNs) trained on large datasets perform poorly in such data-scarce conditions, while large language models (LLMs) generalize well but lack the numerical precision required for low-level vehicle control. This paper presents a hybrid trajectory planning framework that fuses a finetuned LLM with a multi-branch DNN at the trajectory level and applies a Model Predictive Control (MPC) controller as a final optimization stage. Gemini 1.5 Flash, fine-tuned on 479 labeled scenario-trajectory pairs, generates 15- aypoint trajectory proposals from structured textual scene descriptions. A modular DNN processes the same scene as a numerical input vector and produces a parallel trajectory proposal. The two proposals are fused by a learned trajectory refinement network, and the output is passed to an MPC controller that enforces vehicle dynamics constraints while minimizing a multi-objective cost function. Training relies on a three-step distortion- based augmentation technique, curve distortion, scale distortion, and Gaussian smoothing, that generates semi-realistic trajectory variants from a base dataset of over 70,000 labeled rows, reducing the data volume required for effective DNN training. Evaluation across 200 runs in the CARLA simulator on unseen urban scenarios shows that the MPC-enhanced hybrid model achieves an Average Displacement Error (ADE) of 0.66 m and a Final Displacement Error (FDE) of 1.94 m, corresponding to improvements of 37.7% and 34.3% over the DNN-only baseline. The collision rate (CR) decreases from 3.71% to 2.90%, a 21.8% reduction. These results establish that trajectory-level fusion of LLM semantic reasoning with DNN-based feature extraction delivers measurable accuracy and safety gains in data-efficient autonomous driving conditions. The ADE/FDE improvements are statistically significant (p < 0.001). The collision-rate reduction is a consistent trend across ablations, but it is not statistically significant at n = 200 runs (see Section 4.4).