Online Fine-Tuning Policy with Offline Pretraining via Adaptive Dynamic Programming

Adaptive dynamic programming faces a critical challenge in balancing the offline data efficiency with the online adaptability. Existing hybrid paradigms impose stringent requirements on offline dataset quality and diversity, leading to excessive pretraining costs. In this paper, we propose a novel online fine-tuning framework with offline pretraining via adaptive dynamic programming. First, a relaxed linear approximation of the target nonlinear system is used to generate initial policy parameters without strict adherence to original dynamics, significantly reducing offline learning costs. A synthetic data buffer derived from this linearized model mitigates the cold-start problem with sparse real-world data. Finally, an online adaptive dynamic programming method is used to fine-tune the offline pretraining policy through real-time interactions. The online finetuning policy is validated by a nonlinear robotic manipulator to enable effective pretraining and robust adaptability in dynamic environments.