ContextualGPT: Lightweight GPT-2 with Dynamic Routing Architecture for Multi-Turn Dialogue Contextual Optimization

This study proposes an innovative lightweight GPT-2 enhanced architecture that significantly enhances the model's multi-turn dialogue contextual awareness through a dual-branch dynamic routing mechanism while enabling efficient local deployment. Inspired by the Mixture-of-Experts (MoE) concept, we designed the ContextualGPT model. This architecture preserves the core structure of the original GPT-2 while creating a parallel branch for the feedforward network in the final Transformer block, adaptively fusing the different contextual capture capabilities of both branches through a dynamic routing mechanism. Our solution adds fewer than 4% additional parameters to GPT-2-Small's 124M parameters while effectively enhancing the model's understanding of multi-turn dialogue contexts and coherent processing capabilities. Experimental results demonstrate that our dual-branch dynamic routing method operates efficiently in local environments and shows distinct advantages in complex contextual reasoning and coherent generation tasks, demonstrating broad application prospects.