LPad: Automatic driving decision generation framework based on large language model and near-end optimization strategy

The rapid updating and iteration of Generative Artificial Intelligence (GAI) technology makes Large Language Models (LLMs) not only limited to better understanding and interpretation of Natural Language (NLP), but also widely extended to the processing and fusion of semantic information of multimodal data, and gives better performance. Real-time decision-making and the safety of autonomous driving in complex dynamic environments face serious challenges. Although traditional reinforcement learning methods (e.g., PPO) can optimize control strategies, they are limited by low-dimensional state representations, and it is difficult to fully understand the semantic information of traffic scenarios (e.g., traffic rules, driver's intention); while large language models (LLMs), although equipped with natural language reasoning capability, lack direct interface with physical action space. To this end, this paper proposes the LPad framework, which for the first time deeply integrates the large language model with proximal policy optimization (PPO) to construct an automatic driving decision-generation system based on semantic understanding. The framework mainly combines a semantic reasoning decision module, a control strategy optimization module, and a fine-tuning decision generation module, so that the system can execute decisions based on common-sense knowledge while continuously updating and optimizing the proximal policies and semantic decisions. A large number of experiments have demonstrated that the LPad framework has significant advantages over pure reinforcement learning algorithms in terms of generalization and semantic understanding of the scene, as well as in terms of low-dimensional data processing over pure large language models. The proposed framework better realizes the integration of cross-domain technologies and achieves a new breakthrough in the field of autonomous driving.