Enhancing Language Model with Both Human and Artificial Intelligence Feedback Data

The proliferation of language models has marked a significant advancement in technology and industry in recent years. The training of these models largely involves human feedback, a procedure that faces challenges including intensive resource demands and subjective human preferences. In this work, we incorporate feedback provided by artificial intelligence (AI) models instead of relying entirely on human feedback. We propose a simulation optimization framework to train the language model. The objective function for training is approximated using feedback from both human and AI models. We employ the method of control variate to reduce the variance of the approximated objective function. Additionally, we provide a procedure for deciding the sample size to acquire preferences from both human and AI models. Numerical experiments demonstrate that our proposed procedure enhances the performance of the language model.