SJ-ORPO: Alignment of LLMs Using Multi-Task Learning and Self-Judgment

The alignment of Large Language Models (LLMs) is typically achieved using costly data collected from human preferences. Recently, LLM alignment performance has been enhanced by incorporating the LLMs' self-judgment capabilities, emphasizing the potential to replace additional human preference data with preferences generated by LLMs. Iterative fine-tuning methods leverage the LLM-as-a-judge (LLM-J) paradigm, iteratively sampling new data from the model to improve alignment. Since this additional data is generated by the LLM-J, we hypothesize that some of these improvements can be directly achieved by training a single model to function simultaneously as both the LLM-J and the LLM-as-a-performer (LLM-P). To test this hypothesis, we use a base pretrained Mistral 7B model and conduct Multi-Task Learning (MTL) on alignment and self-judgment objectives. Additionally, we propose a data-efficient technique that reuses alignment data as self-judgment data. Our experiments reveal improvements in the model's performance compared to the baseline, which supports the hypothesis of positive interference between the two tasks in a multitask learning setting. We focus on a variant of direct preference optimization, Odds-Ratio Preference Optimization (ORPO), and extend it to incorporate a self-judgment objective, resulting in Self-Judgment Odds-Ratio Preference Optimization (SJ-ORPO). A thorough study of linear scalarization across the two objectives reveals the best performance configuration. Our results with Mistral 7B demonstrate a 5% relative improvement on Alpaca Eval through fine-tuning on multiple objectives, without the need for additional natural language data.