Fine-tuning Semantic Vector Models in Trust Region

This study addresses the challenge of fine-tuning pre-trained semantic vector models to accommodate the specific meanings of proprietary vocabulary in vertical domains, while mitigating catastrophic forgetting and preserving model robustness. We propose a novel approach inspired by Reinforcement Learning from Human Feedback (RLHF) and Trust Region Policy Optimization (TRPO) to adjust the semantic similarities of a targeted subset of words or sentences. By introducing a penalty term into the loss function, we constrain the similarity of output vectors before and after fine-tuning, ensuring that changes to specific semantic distances do not adversely affect the relationships among other terms. Empirical evaluations demonstrate that this method changes semantic distances of selected phrases without compromising the global coherence of the model’s semantic space effectively. The finding points to the potential for improved context-sensitive adaptations in semantic modeling with significant applications to specialized fields such as law, medicine, and technology. This research contributes to a robust approach for fine-tuning semantic vector model in trust region.