Measure what matters: Using an AI chatbot to evaluate the mental wellbeing impact of creative health programmes

• Chatbot exemplifies potential for innovative scalable evaluation of creative health programmes • Chatbot wellbeing scores correlated with WEMWBS scores • Chatbot measured outcomes beyond WEMWBS, e.g. creativity, new skills, social connection • Engagement with chatbot exceeded survey completion and was preferred by participants • Participants found chatbot more reflective of wellbeing than traditional surveys Creative health programmes can enhance mental wellbeing, yet their complexity presents challenges for robust evaluation. Traditional outcome measures may fail to capture the nuanced, multidimensional impacts of arts-based interventions. We investigated whether an AI-powered conversational agent (Kenji) could effectively capture wellbeing outcomes in community arts programmes to provide an alternative evaluation approach to conventional survey methods. We deployed Kenji, a domain-adapted AI chatbot, alongside its text-to-wellbeing scoring model across four arts programmes in London over nine weeks, engaging 73 participants through structured yet naturalistic chat interactions. Participants' wellbeing was assessed using both an AI-based scoring model and the Warwick-Edinburgh Mental Wellbeing Scale (WEMWBS). We examined engagement patterns, correlations between AI-derived scores and WEMWBS, and the breadth of qualitative outcomes captured. Participants demonstrated significantly higher engagement with the chatbot compared to the WEMWBS (86% vs 50% completion rates) and preferred the interaction with the chatbot (89%). The overall AI-derived wellbeing scores showed positive correlation with WEMWBS scores (corr = 0.66, p < 0.05). The AI-based scoring model also captured broader qualitative outcomes, including enhanced sense of purpose (mentioned by 95% of participants), improved mood (95%), increased creativity (93%), and strengthened social connections (86%). Our innovative study exemplifies the unexplored potential of AI-driven tools to enable scalable, person-centred evaluation of creative health interventions. This approach has the potential to address key methodological limitations in arts and health research and to offer new avenues for community organisations, funders, and healthcare providers seeking integrated, evidence-based approaches to social prescribing solutions.