LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information

Recent advancements in large language models (LLMs) have markedly improved their capacity to handle long text inputs; however, current models, including GPT-4o, still exhibit unsatisfactory performance in long-form generation.Generating high-quality long-form content still remains a significant challenge.In this paper, we present LongDPO, a novel approach designed to enhance long-form text generation through step-level supervision.By leveraging Monte Carlo Tree Search (MCTS) to collect stepwise preference pairs and employing a global memory pool to maintain factual accuracy, LongDPO effectively mitigates issues such as inconsistencies that are prevalent in long-context LLMs.Furthermore, we integrate critique-augmented generation to refine the selected preference pairs.Following the collection of stepwise preference pairs, we apply stepwise preference learning for fine-grained optimization.Experimental results demonstrate that our method enhances performance on longform generation benchmarks (e.g.LongBench-Write) while maintaining nearly lossless performance on several general benchmarks.