Forma mentis networks predict creativity ratings of short texts via interpretable artificial intelligence in human and AI-simulated raters

Abstract Creativity is a fundamental skill of human cognition. We use textual forma mentis networks (TFMN) to extract network (semantic/syntactic associations) and emotional features from approximately one thousand human-, GPT3.5-, and Sonnet 3.7-generated stories. Using Explainable Artificial Intelligence (XAI) we test whether features relative to Mednick’s associative theory of creativity can explain creativity ratings assigned by humans or AI raters. Using XGBoost, we examine 5 scenarios: (i) human rating human stories, (ii) GPT-3.5 rating human stories, (iii) GPT-3.5 rating GPT-3.5 stories, (iv) Sonnet 3.7 rating human stories, and (v) Sonnet 3.7 rating Sonnet 3.7 stories. Our findings reveal that GPT-3.5 and Sonnet 3.7 ratings differ significantly from human ratings not only in terms of correlations but also because of feature patterns identified with XAI methods. GPT-3.5 and Sonnet 3.7 favour “their own” stories and rate human stories differently from humans. Feature importance analysis with SHAP scores shows that: (i) network features are more predictive for human creativity ratings but also for ratings by GPT-3.5 and Sonnet 3.7 for human stories; (ii) emotional features played a greater role than semantic/syntactic network structure in GPT-3.5 and Sonnet 3.7 rating their own stories. These quantitative results underscore key limitations in the ability of GPT-3.5 and Sonnet 3.7 to align with human assessments of creativity. We emphasise the need for caution when using AI models to assess and generate creative content, as they may not yet capture the nuanced complexity that characterises human creativity.