Modifying Gesture Style with Impression Words

When people form impressions of others in face-to-face communication, gesture style (i.e. the way of gesturing) impacts their impressions, such as being well-mannered, honest, and enthusiastic. As a mechanism for changing the gesture style, we trained a GAN-based style transfer model using a collection of video clips of speakers. Then, we collected a new speaker dataset from YouTube videos representing three different countries, and applied them to the style encoder of our style transfer model and created a gesture style latent space. However, it is difficult to select an appropriate style from a large number of style candidates to synthesize motions that would give off a specific impression. To assist users with this, we propose a method for automatically selecting an appropriate style by fine-tuning a Large Language Model (LLM) that uses a list of impression words as input. An evaluation study found that the gesture transfer model effectively changes the impression of gesture, and styles selected by the style selection mechanism produced motions that express similar impression to those that applied ground truth styles, compared to randomly selected styles.