Automated assessment of creativity in multilingual narratives.

Researchers and educators interested in creative writing need a reliable and efficient tool to score the creativity of narratives, such as short stories. Typically, human raters manually assess narrative creativity, but such subjective scoring is limited by labor costs and rater disagreement. Large language models (LLMs) have shown remarkable success on creativity tasks, yet they have not been applied to scoring narratives, including multilingual stories. In the present study, we aimed to test whether narrative originality-a component of creativity-could be automatically scored by LLMs, further evaluating whether a single LLM could predict human originality ratings across multiple languages. We trained three different LLMs to predict the originality of short stories written in 11 languages. Our first monolingual model, trained only on English stories, robustly predicted human originality ratings (r = .81). This same model-trained and tested on multilingual stories translated into English-strongly predicted originality ratings of multilingual narratives (r >= .73). Finally, a multilingual model trained on the same stories, in their original language, reliably predicted human originality scores across all languages (r >= .72). We thus demonstrate that LLMs can successfully score narrative creativity in 11 different languages, surpassing the performance of the best previous automated scoring techniques (e.g., semantic distance). This work represents the first effective, accessible, and reliable solution for the automated scoring of creativity in multilingual narratives.

Authors

Publication Details

Journal
Psychology of Aesthetics Creativity and the Arts
Published
2025-03-13
DOI
https://doi.org/10.1037/aca0000725
Citations
12
Primary Topic
Creativity in Education and Neuroscience
Type
article
Field-Weighted Citation Impact
23.30
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Automated assessment of creativity in multilingual narratives.

Rogelio Puente‐Díaz, Sameh Said‐Metwaly, Dan R. Johnson, Claire E. Stevenson et al.
12 citations
Psychology of Aesthetics Creativity and the Arts
Creativity in Education and Neuroscience
23.30
article

Automated assessment of creativity in multilingual narratives.

Rogelio Puente‐Díaz, Sameh Said‐Metwaly, Dan R. Johnson, Claire E. Stevenson, Felix-Kingsley Obialo, Mathias Benedek, Marcela Ovando‐Tellez, Emmanuelle Volle, Matthijs Baas, Wenpeng Yin, Simone A Luchini, John Patterson, Benjamin Goecke, Qunlin Chen, Janet G. van Hell, Boris Forthmann, Aleksandra Zielińska, Ricardo Primi, Roger E. Beaty, Todd Lubart, Izabela Lebuda, Maciej Karwowski, Baptiste Barbot, Iana Bashmakova, Giovanni Emanuele Corazza, Kirill G. Miroshnik, Yoed N. Kenett, Ibraheem Muhammad Moosa
article en
12 citations

Abstract

Researchers and educators interested in creative writing need a reliable and efficient tool to score the creativity of narratives, such as short stories. Typically, human raters manually assess narrative creativity, but such subjective scoring is limited by labor costs and rater disagreement. Large language models (LLMs) have shown remarkable success on creativity tasks, yet they have not been applied to scoring narratives, including multilingual stories. In the present study, we aimed to test whether narrative originality-a component of creativity-could be automatically scored by LLMs, further evaluating whether a single LLM could predict human originality ratings across multiple languages. We trained three different LLMs to predict the originality of short stories written in 11 languages. Our first monolingual model, trained only on English stories, robustly predicted human originality ratings (r = .81). This same model-trained and tested on multilingual stories translated into English-strongly predicted originality ratings of multilingual narratives (r >= .73). Finally, a multilingual model trained on the same stories, in their original language, reliably predicted human originality scores across all languages (r >= .72). We thus demonstrate that LLMs can successfully score narrative creativity in 11 different languages, surpassing the performance of the best previous automated scoring techniques (e.g., semantic distance). This work represents the first effective, accessible, and reliable solution for the automated scoring of creativity in multilingual narratives.

Psychology of Aesthetics Creativity and the Arts
Openalex Percentile: Top 0%
Creativity in Education and Neuroscience
23.30
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.