Can Machines Tell What People Want? Bringing Situated Intelligence to Generative AI

Rather than using AI to make art that imitates the work of human artists, we aim to design machines to support human artists in their creative work. However, assisting in creative tasks without getting in the way of the artist’s workflow can be challenging. Since there is no "correct" answer in creative works for the machine to steer towards, the machine must gauge the "goodness" or "badness" of its actions by watching the user’s response. We present our system, ‘Hero’, for eliciting and developing interactions around creative machines, particularly drawing machines. It uses multi-modal information about the user to steer when it should intervene in the creative process. We are using this system in user studies and interviews with artists to understand what modalities they would use to interact with and give feedback to a machine without disrupting their usual sense of creative workflow. This paper outlines this project’s background, Hero’s current system design, ongoing user experiments, and our planned future work to develop a mixed-initiative, multi-modal collaborative machine for creative tasks. This is a step towards bringing generative artificial intelligence into the messy complexity of physically embodied creative practice.