CAD score: context-aware semantic diversity for scalable creativity evaluation of LLMs

Large Language Models (LLMs) are increasingly recognized for their potential in creative applications, yet quantitatively evaluating the diversity of their generated responses remains an underexplored and challenging problem. Beyond simple lexical variation, creative diversity in LLMs refers to the capacity to generate a wide spectrum of conceptually distinct and contextually relevant ideas in response to open-ended tasks. However, existing automated metrics often operate in a ‘context-free’ manner, disregarding the relationship with the prompt, or, like the recent ‘LLM-as-a-Judge’ approach, suffer from high costs and low reproducibility. To address these problems, this article proposes a new metric, the Context-Aware Diversity (CAD) score. CAD generates context-conditioned embeddings by explicitly combining each answer with the query, and then quantifies the semantic volume of the generated answer set by applying the eigenvalue entropy calculation from the Vendi score. We validated the CAD score on seven LLMs, spanning the Llama-3, Qwen-2.5, and Gemma-2 families, using a benchmark of seven creative tasks based on the Torrance Tests of Creative Thinking (TTCT). Validation results show that the CAD score demonstrates a consistently higher Spearman correlation with GPT-4-judged flexibility scores than all baselines, including surface-level and other context-free semantic metrics. Notably, in a partial correlation analysis controlling for the confounding variable of answer length, existing baselines failed to retain statistical significance, whereas CAD maintained a robust correlation. Moreover, an ablation version that intentionally removed context information also lost this statistical robustness, demonstrating that context-conditioning is essential for meaningful evaluation. Our further analyses also revealed that model scale does not necessarily correlate with diversity, that performance rankings vary depending on task type, and that models exhibit distinct generative tendencies, ranging from exploratory to conservative. These findings suggest that the CAD score is useful not only as a quantitative benchmark for creativity evaluation but also holds practical potential in applications such as a reward signal for post-training or for evaluating candidate responses in Retrieval-Augmented Generation (RAG) systems.