Evaluation Method of Utterance Characteristics via Contrastive Learning

Dialogue systems mimicking characters from stories are useful not only for interacting with users who likethose characters, but also hold promise for supporting story creation. For instance, preparing multiple characterdialogue systems and having them interact as characters within a story can facilitate the creation of scenarios andscripts. Furthermore, writers could use character dialogue systems to explore new ideas and plot developmentsthrough interaction. A crucial aspect of building character dialogue systems is filtering out inappropriate utterancesthat do not align with the character. This paper introduces a novel model designed to capture the character-likenessof a given utterance. The proposed model takes as input the utterance to be evaluated and a set of utterances by thetarget character, and returns a score representing the character-likeness of the input utterance. The proposed modelperforms contrastive learning based on the identity of the speaker. Through contrastive learning, the proposed modelcan evaluate the character-likeness of a given utterance even if the training data does not include data from the targetcharacter. We also constructed a new dataset consisting of 2,155 utterances from 100 Japanese online novels for thetraining and evaluation of this model. Experimental results show that our model outperforms existing methods and abaseline using a large language model.