Rationalize and Align: Enhancing Writing Assistance with Rationale via Self-Training for Improved Alignment

A Writing Assistant (WA) is a system that offers writing suggestions based on user instructions.Existing WAs are typically built by training large language models (LLMs) on domain-specific instruction data through supervised fine-tuning (SFT) only.However, SFT optimizes models to match a single reference, failing to capture the inherent flexibility of text editing, where multiple valid revisions exist.Therefore, solely relying on SFT limits WA performance.To address this limitation, we propose the Rationalize and Align framework, which enhances the WA performance with rationale (i.e., linguistic explanations) and alignment.Our framework automatically generates the rationale and preference data for writing tasks via distillation and self-training, eliminating the need for human annotation.These data are then leveraged to refine WA using a novel preference optimization method.Empirical results show that our framework significantly improves WA performance.Our WA outperforms both open-source state-of-the-art WAs and the closed-source GPT-4o by 3.9 and 7.1 points on average, respectively, across eight well-established writing-related test sets. 1