Creative Evaluators: When Machines Evaluate Us

Abstract The Computational Creativity literature stresses the importance of evaluation in computational creativity systems in addition to generative capabilities. But, what about machines that only evaluate? When it comes to co-creative systems, humans often take on the primary evaluative role, while machines assist with the generation of creative artifacts. In this paper, we propose flipping the paradigm, envisioning machines that (only) evaluate humans in the creative space. We call such machines Creative Evaluators whose end goal is to evaluate human-made artifacts and provides feedback to aid humans in the creative process, while the machine itself refrains from directly generating any content. We present a co-creative framework for designing Creative Evaluators, which includes the tension between originality and quality of creative artifacts, and the need for explainability and fairness in machines that take on an evaluative role for human-made artifacts. To illustrate this framework, we present Titlevator , a machine that evaluates titles for AI papers, while also providing feedback to the user on the originality and the quality of the paper being evaluated.