A Comparative Study of LLM-used Intention Estimation Methods for YES-NO Question Answers including Unknown Answers

In dialogue systems, it is especially important to ask questions that can be answered with either “yes,” “no,” or“I don’t know” intentions (YES-NO questions) in order to confirm the user’s intentions and status, and to accuratelyinterpret the intentions of the user’s answers. In this study, we aimed to perform highly accurate intention estimationfor generic topics, and to be able to determine unknowns as well. Specifically, we created a question answering corpus(Japanese yes-no question-and-answer pairs), designed multiple intention estimators using a large-scale languagemodel, and compared and evaluated their accuracy. As a result, the GPT model with an additional all-combininglayer (Fine-tuned GPT) showed the highest estimation accuracy, achieving 91% accuracy. On the other hand, whenprompt programming was performed using GPT-4 (Few-shot learned GPT-4), we observed a possibility that there wasa tendency to judge as unknown responses for which intention estimation was difficult. The results of this study areexpected to provide valuable guidelines for future research and practical use, as they suggest a policy for selecting anestimation method and tuning a model in the intention estimation task.