FROM COMPLEXITY TO CLARITY: ONE-STEP PREFERENCE OPTIMIZATION FOR HIGH-PERFORMANCE LLMS

Large Language Models (LLMs) have transformed natural language processing, achieving state-of-the-art results in text generation, reasoning, and problem-solving. Despite these advances, aligning LLM outputs with nuanced human preferences remains a challenge, hindered by the inefficiencies and instability of traditional reinforcement learning (RL) methods such as Proximal Policy Optimization (PPO). These multi-stage pipelines often introduce high computational costs and degrade core model capabilities. This paper proposes two unified RL-based algorithms, Odds Ratio Preference Optimization (ORPO) and Group Relative Policy Optimization (GRPO)which combine supervised fine-tuning (SFT) and preference alignment into a single training phase.This integrated approach eliminates the need for separate reward models and sequential stages, significantly reducing the risk of catastrophic forgetting while enhancing training efficiency.Empirical evaluations on Mistral-7B and Llama-3-8B across six benchmarks (MMLU, MATH, GSM8K, HumanEval, and TruthfulQA) show that ORPO outperforms PPO, achieving a 23% improvement in reasoning tasks and a 37% reduction in training time.Lyapunov-based theoretical From Complexity to Clarity: One-Step Preference Optimization for High-Performance LLMs https://iaeme.com/Home/journal/