Diffusion Model Dynamic Trajectory Splicing Decision Transformer

Offline Reinforcement Rearning (RL) faces critical challenges in balancing policy improvement with distributional shift, particularly when leveraging trajectory-based models like the Decision Transformer (DT). This paper introduces a novel framework that integrates uncertainty-aware action generation and conservative value estimation to address these challenges. A diffusion model dynamically adjusts the number of candidate actions based on environmental uncertainty, expanding exploration in high-risk states while maintaining efficiency in predictable scenarios. The framework further employs an Implicit Q-Learning (IQL)-based critic network to filter generated actions through a dual mechanism: discarding out-of-distribution actions via a value threshold and selecting the optimal action through conservative Q-maximization. By synergizing trajectory modeling with uncertainty quantification and value constraints, our approach effectively mitigates distributional shift while preserving the flexibility of policy learning. Extensive evaluations on D4RL benchmarks demonstrate the framework's robustness across diverse datasets, from suboptimal to expert-level trajectories, and highlight its potential for real-world applications where safety and adaptability are paramount. The source code is publicly accessible at https://github.com/Beardoglxh/DFDT.