Lessons on Facial Reaction-Reading for Bantering Robots: Toward Next-Level Crowd Work for Robot Comedy

Despite the modern proliferation of artificially intelligent technologies that use language to interact with people, social robots, AI assistants, and other related technologies have a remarkably bad ability to playfully use humor while processing contextual social information about their surroundings. For example, the ability of these types of systems to read nonverbal social cues from their surroundings and use this information to enhance interactions is mostly absent, and when it does appear, it is usually in controlled research settings. At the same time, these types of bantering skills support the success and engagement of skilled humans in domains from entertainment to service; accordingly, in our work, we seek to equip robots with better abilities to read users’ faces after humorous quips and to use these facial responses to shape next interaction steps. Toward this goal, we surveyed human comedians to understand their methods for visually “reading the room,” applied/tested the observed approaches in a beginning visual classifier, and applied/tested an improved face-reading method based on the results of our first experiment. The results show that our face-reading classifier improved over the course of the experiments and ultimately achieved close to a human-level performance. Repartee in general tended to enhance the interactions. Overall, this work can benefit socially interactive technologies by equipping them with new nonverbal behavior-reading skills.